Kyle d'Oliveira
Appearances
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
to the size where we can exhaust regular integers and we need to actually go into like beacons. We look at approachability, how easy can we just take a new developer and dump them into the code base and have them up and running?
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And it also removes a lot of this argument over should I use double quotes, should I use single quotes? If it just automatically fixes and developer can write whatever that they want, that's fine. But I've also run into issues of having pull requests being bombarded by style, and it really distracts from the code review about the behavior.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Yeah, definitely need to be careful there.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
So actually, it's not clear where that point exists. I think it's probably going to be different for every organization and probably different for exactly the work that you're running into. I think the thing is to listen to the pain points of the developers. So if you notice that there are, you know, there's pieces of friction that occur between developers, like...
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Because as things go to scale, there's obviously new patterns that need to be adhered to that we don't necessarily need to focus on with small projects, but we do need to focus on for large projects. And my team has focused a lot of that to making the effort and experience for all of the developers easy and fast.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
That's the point where maybe there's actually some tools that need to be built to make this easier. So one thing that I think comes up really quickly in organizations is often the concept of a testing server. So you've got your developer's environment, you've got maybe your CI, but maybe you want a production-like environment for things. And so you have a staging server.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
When there's five developers, it's really easy to just coordinate and be like, oh, staging is mine now. I'm going to test something. When it's done, I will hand it off and maybe reset it back to whatever the master branch and let people work that way. But that really falls apart when you have 100 developers.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
How do you coordinate one server where everyone is trying to test something if you have 100 developers fighting for that resource? You can kind of... fudge it a little bit by maybe having a fixed number and you round robin them out. But again, at some point, that's going to break down.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
So if you think about what's the problem here is that every developer wants to potentially test something on an asynchronous schedule Maybe it actually makes sense to build some tooling so that you can spin up staging servers on Amazon EC2 or on Google on demand and just route them there.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And so that's something that we ended up having to do really early of building our own tooling so that we can, we call them beta environments where we can have arbitrary number of them. Someone spent the effort to basically say like this branch on GitHub, I want a clone of the site.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
on amazon and within like 10 minutes you've got a domain that points to it you've got the full stack you can you have full control you can do whatever you want you can break it and it gives developers a lot of autonomy to test things that they want and you know removes a lot of this oh let's deploy it and see what happens you have a full environment that you have full control over go test it go see it with as much data as you want and then see what happens
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Another example kind of along those things is like deployments. Do you have a handful of senior developers who can deploy or do you deploy on like a every Monday you do a big deployment? Like that's going to start really breaking down when you have a lot of developers. You know, at Clio, we everyone has the ability to deploy. Everyone has the ability to merge code.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
So we give the power to the developers and now, you know, A junior developer can come in, write a fix to a readme, merge the code, deploy it without having to really bother people outside of getting a code review. And now we're deploying code probably upwards of 30-ish times a day, and that number is just only going to go up. And...
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
So as we're running into these issues, we are just looking at what can we do to build tooling so that it's no longer frustrating for developers. And the important part of this is developers need to voice things and managers and companies need to listen. If we're wasting five hours a week per developer on this one thing that's frustrating, build tooling around it.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
It used to be something that we did. We used to call it the snapshot, and you could point environments at the snapshot and run test queries on it. But we actually did hit a size where the time it took to set up the snapshot every day was taking longer than it would take to actually...
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
back it up so we it was just starting to become unfeasible for us and we're also dealing with sensitive data and we don't necessarily want to give free access to all of that that data for our our clients so we instead try to invest in a little bit of tooling we we definitely still have issues where everything looks good in development everything looks good in like beta or test and we deploy to production something is wrong so we think about what can we do to make make that better
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And so we, you know, if it's about a lack of index on like a database query or something like that, we can try to check that ahead of time and build some tooling and alert people when something goes wrong. But also in production, we can be and say like, hey, this query took 30 minutes, that's unacceptable. This query took five minutes.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And return that information as like an exception to the developers that they need to fix, but without interrupting the actual request behavior. And if things go really south, just roll it back. Like we don't really have, it's not a blame if someone deploys something and goes south and they quickly roll it back. We just try to take that as a learning opportunity.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And how can we take that learning opportunity and share it to everybody so that everyone learns from it? Does that answer your question?
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Yeah. What we can also do is, I mentioned earlier that we could talk. We have these data environments that we can spin up. You just use a SQL dump to store data in there. And although this isn't necessarily production data, developers have full control over what that data looks like.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And so if we wanted to see what happens if there is tens of thousands of something in a table or more, we could just build little scripts that can seed that database and then test it outside of production.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
It's not perfect because it doesn't always match the same shape as production, but it's an iterative process and that information gets codified so you can keep adding to the seeds in those manner so that it becomes a better and better representation as we go forward.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Yeah, metaprogramming is actually one of the best strengths of Ruby. You can do so much with it, but once you have it, it's the hammer and everything is a nail and you want to use it. And that's often a trap that new developers, when they learn about metaprogramming, they really want to go into.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
I think a good lesson to come out of that story is that if you think about code, it's written once but read countless times. And so if you can take little things to optimize the code for the reader, that is much better than sacrificing readability to optimize for the writer.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
So if it takes you an extra 30 minutes to write a whole bunch of cookie cutter methods, but now those methods are in place and they're static and it's easy to read and reason about and test, that is well worth that 30 minutes because you're going to lose more than that reading that piece of code in the future.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Yeah, absolutely. Anything that you can do to make those kind of classes easier to understand and read for a new person is great. And also just backing up a little bit to your example, this is also an instance where metaprogramming bit you, but metaprogramming is also interesting that it could save you because you can also ask Ruby about Ruby.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
So if anyone didn't know, this is a tactic that I use all the time for debugging pieces of code that I've never been familiar with. If you can have access to a console, you can ask Ruby what methods are available with a .methods call. You can also get access to the method itself and then ask it, what is its source? Where does it live?
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
make life easier to track down methods that may be dynamic or created by gems.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
I think that's another example of someone making some tooling that makes something that, yeah, if you knew to call .methods and subtract object.new.methods or object.methods, it's great, but now it's two characters, and it's nice and easy, and it's much more approachable, and you can have access to things that you may not knew existed.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Oh, most of my career has been working with Rails. So before Rails 0, I was working at an enterprise Java shop that I don't remember a lot of the details of it anymore. It's kind of too far in the past. But I think I've been working with Rails now for 11 years, I think. So it's been just a long time with just Rails. I don't remember a lot of the pre-Rails world, to be honest.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
We've definitely been bitten by how easy it's been to make N plus one queries in the past. I think pretty much any Rails shop is going to be doing it. Rails offers tooling to help with that, but the tooling still requires a lot of effort. You have to kind of know what N plus one query you're introducing and fix it.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Yeah, absolutely. It's hard to know what you don't know. And so if you don't think about the scale at that point in time, it's hard to know what problems you're even going to run into.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
So though that's where you can build some more tooling, there exists a gem that we built, the JIP preloader. There's also another community gem called Goldiloder that removes stuff like N plus one queries. And those are ways to basically eliminate those kind of problems. Some other things that kind of come up on Rails as we are building is discoverability of templates.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
So I think one of the previous episodes of Ruby Rogues was talking about this. But as it scales up, Rails ERB makes it really easy to render partials all over the place. But it's really hard to understand, if you're looking at a page, where are those partials actually coming from? And how can you dig back into them? So that's a challenging thing with Rails as well.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
There's also some things with the community for things like paging that can be problematic at scale. If you look at what some of the basic gems that offer, it often comes down to a limit offset, which
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
is also really fine on small datasets, but as you get to datasets that are really, really large and you're going to page really deep into them, it actually starts really falling apart and breaking down and things that you might not know until you actually just hit that scale.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
I think some of the Rails conventions also starts becoming a little bit problematic, and you see a little bit of discussion about this. Rails at one point said, throw all the logic into the controller. And then eventually the controllers became skinny and all the models became really fat.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And I'm sure everyone has that God object that exists in their project, the user object or the account object that is 5,000 lines and really difficult to reason about. And people are offering opinions of having like service classes or various different patterns to try to combat that. But we're still trying to unpack some of the things that you started the Rails projects with.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Yeah, and here's a little bit of my thoughts from it, but I'm not going to say my thoughts here are perfect. I think everyone's mileage will vary because documentation is a tricky thing. So when you get to... If you're getting to... gotchas. If you ever tell someone, oh, if you see this pattern, don't do it.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
If you have code reviews that are like, oh, I've been bitten by this before, that should be something that falls into the linting or the just-in-time education where you try to codify that.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
If you see people that have inline comments in code that says, you know, like, this next few lines are going to iterate over something and do these operations, that's probably an indication that their code is not written well to describe it, and that comment is not super valuable.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
So that actually might be something of like, that comment shouldn't exist, and instead we should maybe extract a method that describes it better and kind of move in the direction of code describing itself. When you are implementing something that's specifically tied to code, it should probably exist at the code level.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Yeah. So working at Clio, the code base is quite large. We have tens of thousands of commits that we go through. And it's really easy to see patterns of developers working on features. The features go live. And at some point in the next six months, a year, those features come back to bite us. So as like the first commit is great, the 10th commit is great. You're starting to notice some things.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
So if you have a module that you want things to include and developers need to implement certain methods in there, maybe the module should... define those methods and raise a not implemented error that have a very clear, this is what this method should do, this is what it should return, here are some examples, and just link to them in your own code base.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And so now when a developer looks at that specific piece of code, it's still tied to the code base. But all of that's you know, at the code base level, there still needs to be something at like a higher level. That's like a readme in the documentation or in something else entirely.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
So we have stuff that exists in a readme that's kind of more about like process, but process is specifically related to our code base. So a good example of this would be, how do you do these asynchronous migrations? So this isn't really super tied to code because you might make a migration, but then what's the process of getting that live? So we have a step-by-step guide for at Clio.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
If you want to do a migration, here are the steps that you need to take. And as much as we can, we just link back to code rather than re-implement the code, but we'll also just describe things in English and offer templates there. And then we go one level higher to things that exist more at like a process level for the organization. So for that, we use a tool called Confluence.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
There's lots of tools that exist that kind of do similar things. But for those, that's things that exist outside of the code base. So if an incident happened, how do you do a postmortem or root cause analysis on that? And there'll be documents for that.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Or, you know, if you wanted to propose a new style of new feature that you wanted to get some buy-in using some new architecture, just wanted to make sure that the approach is correct. You can do like a design doc in Confluence and get people kind of bought in well before you've actually written the code. But once the code is written, that document is less relevant.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Yeah, so there's definitely things that linters aren't going to be able to do. A linter won't be able to tell whether this thing should be a model or a service class or something. It's not really going to be able to... It doesn't understand the business logic of it. So for things like that, we kind of have to rely on little handbooks of being like, here, we've codified our style guide.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And we try to make sure that we keep that up to date. There are some things that we still teach through kind of tribal knowledge and code reviews. Like if someone submits a pull request and we notice it, we'll still correct it there and we'll do a lot of pairing. So we'll get developers up to speed by working with people as opposed to just going off on their own.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
But I think this is just a learning process. Like we, I don't think we are perfect at getting developers onboarded and I don't think anyone is. And I think that's the important distinction. It's an iterative process.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
If you bring in three developers and they all have the same issue, that's probably when you might need to introduce some new documentation and be like, hey, here's our new developer handbook. You might want to read it.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
So I think the answer is both. I definitely think we still have places where our documentation drifts and then somebody notices and we're like, oh shit, we got to fix that.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
But we also do leverage tools like Danger, Danger.js, like GitHub, where it can look at code and it's not necessarily like a linter of basically saying like, hey, this is bad, but it can make a comment of being like, hey, you're doing something. Maybe this is related to this link over here and direct developers or whoever's reviewing to go take a look at the documentation.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
By the 100th, there's maybe some problems. And by the thousands, thousands commit on it, right? You've stopped because now you have to completely refactor and rebuild a lot of this technical debt that you introduced. So my talk was talking about some of the lessons that we've learned.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Maybe there's no changes that require there, and we definitely need to be careful about how much noise we generate. But in the case of a migration, if a developer writes a migration and then submits it, we could basically say, hey, did you add a new file to the db migrate file?
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
If so, make sure you're following the steps in here and make sure that it aligns and kind of point them back at the documentation, both for the writer of the pull request, but the reader. And then kind of helps make sure that things stay in sync. Not a perfect process. I think we're slowly getting better at making sure the documentation stays up to date.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Yeah, actually, we are in the same situation. I think we were going to hire a DBA this year prior to the pandemic, and then I think there were some complications. But prior to that, the company has been operating for over 11 years, and I think now no DBA.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
We definitely have some DevOps that are a little bit focused on making sure that the database is running and making sure that we've got replication set up and proper statistics. But we kind of put the onus on everyone. Like you don't have one person who is the guru of SQL. You have everyone. And so everyone tries to teach everyone these things.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And we try to do our best to share that knowledge where we can to make everyone as experts as we can. So we've managed to go 11 years with no DBA, and I think we're only getting to wanting one now because we're trying to do really customized processes of this online schema migration stuff.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
How do we make that completely automated, which is actually going to be a completely distinct system to the Rails system because we're going to want to apply it to any of our projects. Or maybe some gotchas between upgrading MySQL. There's probably some things that they might actually have really good insight into.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And although the lessons are very specific to specific problems, there's kind of a generalized idea of what some approaches that you can take to dealing with technical debt in your own projects. If you are able to, for instance, keep if you're able to automate technical debt away entirely, well, there's a whole classification of problems you no longer need to think about.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
But I think our general approach is, even in that situation, we're going to have one DBA and hundreds of developers. And we want to make sure that they may have knowledge that might be useful for talking through things and sharing things, but the work is going to still fall on the developers.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And we need to make sure that everyone is learning as much as they can and not just blindly hoping that the DBA is going to handle it.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Yeah, I would agree. And I think there are more specialized roles, but I think there are skills that apply to everyone. So, you know, as the company grows, you may have more specialized roles that have more specific knowledge, but I think probably with that specific knowledge comes the responsibility that they are not gatekeepers of that knowledge, right?
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
They may be experts and they're maybe building content, but I would say part of their job is to make sure that that content is consumable by everyone. And if they're answering the same questions over and over and over, they're not doing their job to educate people on how to self-serve and do it themselves.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And that's how we learn and grow as a community and get better is just by sharing this knowledge.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Sure. My name is Kyle. I've been working for a company named Clio. That's a legal practice management SaaS software. It's based out of Vancouver, Canada. It makes... Practice management software aimed at lawyers. We're looking at transforming the legal space. Our mission is to transform the practice of law for good. There's a nice little double entendre there.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Yeah, I agree. You don't necessarily have to have everyone customly building SQL. Active Record does a pretty good job of being an ORM that lets developers just do the things that they need to do. And there's data notifications available to easily add tooling that you don't need the DBA for.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
But as things grow, there's things that Rails doesn't yet have tooling, and maybe that's something that if you have a DBA who is well-versed in Rails, maybe they can contribute back to the framework or add their own gems that can help everybody get better at working with databases.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And it doesn't necessarily invalidate their job, but their job becomes more of a knowledge producer, and they try to share that knowledge and make the community better.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Yeah. And I think that's, that's like the roles of the specialists, the people who have the specialized knowledge, they're probably more consultants. And, you know, you have someone who's like, I've got a really gnarly problem. I don't know what to do. Yeah. Like let, get them to like sit down and help with you. And that's a big asset that they can help with people.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And you can feel confident that those are just automatically protected. And if you are cleaning up after yourself as you go and making it easier when there are curveballs being thrown at you, fixing technical debt and dealing with it when you hit scale doesn't have to stop you entirely. It just becomes a constant small tax that you pay.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And, you know, if that's a one-off, it's a one-off, but if they do this 10 times in a week, maybe there's education there, or maybe there's tooling. And I think it goes to pretty much any role that you ever feel like you're just throwing something over the fence. If you push that responsibility also to the developers, you can also end up with a much higher quality project.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Or use some of the gems available and have the N plus one queries just automatically avoided for you.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
I was just going to mention one thing about includes, because I think this is another gotcha of Rails, is they don't really teach you what happens with includes. And includes actually does two things in the background. It either uses a preload or an eager load.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And a preload splits it off into a different query entirely, where you do something like select star from table, where ID is in this big list. But then there's eager load, which tries to smoosh it into one big query. This is something where Rails always suggests using includes because it'll handle that distinction for you. But that distinction actually makes a difference at scale.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And when you're dealing with large tables, eager load is almost always worse.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
significantly and so it's almost all the time you actually want to use preload same same interface but it's just this interesting little gotcha that you don't really realize until it starts biting you and you got to remember everything is just a tool and you can either smash your finger with that hammer or you can build what you want to build with it exactly right kyle well if people want to follow you and some of the stuff that you're doing online where should they go
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
I don't really have a huge online presence. I do have a GitHub account, but that's mostly working on either public gems for the company. But what I'm trying to do is be a little bit more present in the community. So I do have some talks available at RailsConf. And my goal is to be pushing out a little bit more written content, which is available at the blogs that Clio provides.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
So I can provide a link for that in the future, as well as a link to any of the talks that I have. Unfortunately, I'm not a super user on Twitter, but I can also provide my LinkedIn where I sometimes post new information there as well.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
But if you invest in the tools, you can actually start moving faster even as you scale.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
I didn't prepare anything, so I actually don't have anything that's off the top of my mind here for things to just call out.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
I'm always happy to like building tools to fix these things so that we can make the community better.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
I would say technical debt is like accumulation of decisions that are made while coding that you eventually need to correct in the future. And as developers, I think we're always making these decisions. Can we cut a corner here to deliver a feature out a little bit early? And I think those are like technical debt isn't bad.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
I think when you are willing to get something in front of the users and deliver value earlier by incurring a little bit of this technical debt that you then have to clean up, I think that's totally okay. But I think technical debt often comes in the situation of Developers making a decision that a framework needs to be super generic and it's a little bit speculative.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And then they come to implement something in the future and it's just really difficult to deal with because it's so generic and hard to understand that new developers have to then unpack that and wind it back just to implement something new in it. Some things that I think are not necessarily technical debt can kind of come from...
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
maybe decisions that actually made sense at the time and aren't necessarily any cutting a corner. So, I mean, it may make sense to build a system that is very generic and maybe that is the correct choice and you build through and then things change. And when things change, that's when you might have to have the technology that comes back. But until the things change, it actually might not be.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
I think that's a bit of like a generic answer, but it's hard to pin down a concept like technical debt because almost everything we write is debt of some form.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
One of the things that popped up is actually something that we decided on because the Rails community pushes for, and this is what comes out of the box.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
So if you think about Rails migrations, if you think about how they're often applied, if you think about some examples that you've worked on, there are often times where you use something like a tool like Capistrano, which deploys some code, and as part of the code, database migration gets run. And for Projects, That's fine.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
That's like a for most small things like that migration that runs is fast and it's not a problem. But so this is an example of a decision that we kind of were like, let's just inherit what the community uses. But as we started scaling out, we started encountering problems with it. So, for instance, a table that if you ran a migration on it took 30 minutes.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
This means that our deployment took 30 minutes and also timed out. So we lost all of the context of it. But also during this period of time, the table locked. So any developer or any queries that started going to that table stopped being answered. So all of our servers shut down. And we couldn't kill the alter table because it was already mid-progress. And...
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And it's been really interesting seeing some of the changes in legal that we've kind of made an impact with over the last few years. I've been working on Ruby and Rails for the better part of the last decade, but when I started working on Rails, it was Rails version zero, and I've been upgrading Rails ever since, and so now finally up to Rails 6, and...
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
After it finished, we now had a table with a new state, but the code hadn't actually finished deploying. So now we're running into different problems. So this is a little bit of a decision that makes a lot of sense when you're small. Go really quick because you can, and it makes sense. But when you hit a certain piece of scale...
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
you can no longer run with those assumptions and you need to change those. So a new process needs to be built. And for database migrations, we need to build them in a way that are like entirely asynchronous to a deployment process.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Yeah, I think this table that we run uses a little bit of all of the activity that users do. And it was like the first table we ran into that it exhausted like 32-bit integers and we needed to flip the IDs to be big ins. We didn't think that would be a problem either. And it's leaps and bounds bigger than any of the other tables we have in our system.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
There's actually this good question, and there's actually a lot of tools that exist that we don't necessarily need to build ourselves. GitHub has a tool called Ghost. There's another tool by Percona. It's in the Percona toolkit. I can't remember. It's like maybe online schema replacement. I can't remember the exact name.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
The general strategy is to, instead of changing a table with like an alter table, you actually create a brand new table, populate that table with various mechanisms. Some of them use triggers, some of them use the binary logs, get the table to like a table that's in sync, and then do quick renames.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And so you rename the table to be the old one to be old, you change the new table to be the new one, and then new queries start flowing into this new table. And you can do this what As long as you want, it's entirely non-blocking, but it has to be in a process that exists entirely outside of the deployment stack.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Some of them seem obvious. Like I think if you're dropping a column or adding a column that could potentially lock, but some of them are not. Like if you changed a Varchar from like a Varchar 100 to Varchar 200 and you're just increasing it, does that lock? Maybe. I actually don't know off the top of my head. What if you change the character set? What if you change the coalition? I don't know.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Is this on MySQL or Postgres? This was in, we used Percona, which is just an offshoot of MySQL. So it'll also be different for between databases. So Percona might have different decisions.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Yeah. So a lot of this comes from tooling and education, right? There's as like senior developers or people who have just different experience from different places, we've accumulated huge amounts of knowledge and it's kind of all tribal.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And I think the, if you join a company that doesn't have a great strategy, a lot of the strategies for sharing that knowledge is like just work together, go submit pull requests and have them code review it and learn from the code review. And I think that's okay. You can learn that way. But there are better ways to push information to people. And this is a concept about just-in-time education.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
So touching all of the major versions, my major focus at Clio, which I've been at now for eight years, has been on the backend infrastructure side of things. So the main focus is scalability for the code base, but also in the terms of the organization, like what happens when we have 200 developers working? What happens when the data set sizes are, you know,
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
So an interesting example of this can be through the linters. So I did a talk about this as well for the 2020 Couch Edition of RailsConf called Communicating with Cops. that focused on using RuboCop as a mechanism to provide education. I did a little bit of deep dive into how RuboCop works and how to build your own custom comp.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
But one of the things that we approach with at Clio is as people make mistakes and learn about bad patterns, we try to codify those patterns so that it's... It doesn't happen again, but people get education about it right as it happens.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
A good example of this that is super-trivial and doesn't often bite people until there's just an unexpected case would be maybe the Rails convention of naming files. We've seen cases where people maybe make a user's model, but then make a typo in the spec. So rather than call it user spec, call it users, and it's plural, or something along those lines.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And the spec will still run, but there might be some tooling that we expect to adhere to the Rails convention and it doesn't quite line up. So you can have a linter that basically checks the name of the files and the name of the classes and make sure that they're in line. And if not, alert people and do that as part of their editor or do that as part of them committing code.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And they get warnings and they get education as they're writing code. So if they just wrote something, they save the file, they get a little warning popped up being like, hey, you may have made a typo here. And this goes even to as far as behavior.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
If we know that there exists bad patterns, so for instance, making an HTTP call inside of a transaction, which we know is going to be potentially bad, we can actually automatically prevent that. And as soon as that starts happening, as soon as we're able to detect it, so it might be in a test,
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
might be as part of a linter, we provide that education right back to the developer so that they understand what they did wrong and the avenues of what they need to do to fix it. So now when a junior developer enters the company, they can actually just feel free to start writing code and write even code in kind of a way that maybe breaks some patterns.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And a lot of times they're going to start getting education right away. And then we can do all of the usual things as well as as pull requests come in, we can review them and provide more education that way. And if we find constant patterns of every junior developer we come in makes the same mistake, let's codify that so that they get the feedback immediately.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Yeah, absolutely. And I think there's even one step farther of a lot of linters can do auto-correcting. So if you care about having one line space between methods, don't even have RuboCop or a linter warn about that, just auto-fix it. That's something that a developer just doesn't need to worry about.