Dave Kimura
Appearances
Ruby Rogues
Developing your development - RUBY 649
Right. That's pretty cool to be able to block out your colleagues with a click. If only I had that work today. Is there an API?
Ruby Rogues
Developing your development - RUBY 649
So this is operating off a Visual Studio hook, is it?
Ruby Rogues
Developing your development - RUBY 649
No, keep it out. That would be my excuse. I'd be like, why aren't you doing any work? Oh, it's in BIM. It doesn't work in BIM.
Ruby Rogues
Developing your development - RUBY 649
I like to be demoted. Dave has just made his background of his room go green and red in synchronicity with what he was saying. That was very impressive. Yeah. That's automation.
Ruby Rogues
Developing your development - RUBY 649
fluctuating it up and down is probably distracting but just going in stress level yeah like it's red it's red i'm not doing good oh no get it back to green yeah that's that doesn't actually help with coping do your lights go red when your tests go red they what tests
Ruby Rogues
Developing your development - RUBY 649
I think I'd rather be Hugh Jackman, really, wouldn't you? Dude at 95. I think I'd rather be there. This is the image I have of myself. 3 a.m., cigarette in one hand, mixed drink in the other. Maybe two keyboards and a laptop top left. I mean, crikey. Now that I've learned that sensible office hours are the key to programming greatness, I'm kind of reconsidering my career, really.
Ruby Rogues
Developing your development - RUBY 649
I'm going to become rich and famous, rich and more famous, after I create a YouTube plugin for VS Code that enters random keystrokes. There you go. Okay.
Ruby Rogues
Developing your development - RUBY 649
Dude, I've been trying to use the Rubyist app on the iPhone. Do you have this app, the Rubyist app? I don't know. So this is, oh man, I've forgotten who wrote it. It's one of the, pretty sure it's one of the Japanese guys that everyone knows. Anyway, this is Ruby on your iPhone with API hooks into things like Siri.
Ruby Rogues
Developing your development - RUBY 649
So you can kind of automate stuff from your iPhone and you can also kind of write Ruby on the iPhone, right? It's using the mRuby interpreter. So that's how they've got it in there. They've just kind of linked the whole of mRuby into the app. And I'm getting absolutely nowhere with it. I have not got it to do any. And it's me. It's not the app. It's definitely me.
Ruby Rogues
Developing your development - RUBY 649
But it's a fantastic thing to play with when you're waiting for stuff to happen. So that's my pick for this week. It is the Rubyist app. It is free on the app store. Awesome.
Ruby Rogues
Developing your development - RUBY 649
I got a question for you here. So as I'm understanding it, at software.com, you've got loads of really cool tools that can kind of look at people's commits or some other metric you're looking at to assess productivity.
Ruby Rogues
Developing your development - RUBY 649
You've done all the work in that one commit at the start.
Ruby Rogues
Developing your development - RUBY 649
I went on Code Climate the other day. It wasn't my idea. My colleagues did. And it gave my code a D. And quite honestly, that kind of put me off all of these kind of code assessment tools. I'm now a bit scared that if people go to software.com and show them my work, they'll say that, you know, Luke's only productive on like a Tuesday morning. And even then he doesn't do much work.
Ruby Rogues
Developing your development - RUBY 649
It sounds like a great tool for managers. Is it going to help the software grunts, the frontline coders like me?
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So has using Sorbet made you guys write better code?
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
Yeah, real quick. If people want to follow some of the things that you're doing online, where should they go and look?
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
Awesome. Yeah, and I'll go ahead and kick us off with some picks. So my first pick is the wise, thin clients. So I've been getting my kids ready for the digital learning in the upcoming school year. And instead of giving them each a... $600 computer or something, I have deployed a thin client set up at home, which is probably way overkill for most home networks and stuff.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
But I would much rather replace a $100 all-in-one thin client than a $600 computer and monitor. So that's my first pick. And second pick is I, on this call actually, I got the announcement that I got the Apple TDK. So the developer transition kit with the ARM-based processor. So I will be releasing some Drifting Ruby videos on developing on ARM-based processors soon.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
And that's crazy. I think a lot of us would dream to get to that kind of scale. Honestly, I think it would be a nightmare. Not only do you have this super huge code base to maintain, but then just the infrastructure alone and what that would look like would just probably be really crazy. Can you speak into the infrastructure?
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
Are you guys using a very simple infrastructure or has it kind of gone from simple to very elaborate?
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Yeah, absolutely. Although you do have to be careful about the auto-correction. I remember one time in my earlier days of development when RubyMind came out, I tried out RubyMind's code refactoring thing. I forget what they call it. But I had some really poorly written classes, and it just absolutely broke everything.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
I have no idea how that happened, but things just were not working the way they were before. I had to pull that merge back out because, of course, as an early developer, I didn't have any tests on the application. So I didn't really notice that things were broken until they got deployed.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
So you also previously mentioned about, so not necessarily onboarding developers, but having a lot of developers work on the project. So what point do you go from a small shop to a large shop where you have to start putting different kinds of practices in place? And what are those kinds of practices when you're dealing with a lot of developers on a single code base?
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Yeah, that's one of the things that I did just for my own hobby project and just continual learning. I have a self-hosted GitLab instance, and I set up a Kubernetes server, which will automatically create the infrastructure for the application that got pushed. So it always happens on any kind of development or master branch push, and then also on each commit.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Yeah, absolutely. One thing that kind of rings true is you always have to think about scalability when you're developing, but don't actually write for scalability when you're developing. So keep it in the back of your head saying, is this going to come back and bite me later? Or is it a really non-issue?
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
up to the repository, and it'll spin up an entire infrastructure within Kubernetes with an FQDN that that feature can then be tested. So it works on smaller applications.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
I don't know how it would work on applications that consume 30 gigs of RAM of resources, but I think on smaller applications, that kind of thing can really save you from having to have dedicated test servers that's shared by several people.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
I do have a Judgement Ruby episode on Kubernetes, which that's where I got the inspiration from on that episode. I just didn't tie it into the CICD portion.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And we have a special guest, Kyle D'Oliveira. Did I say that right?
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
I remember one time I had a situation where I was storing just three kilobytes of data in a database. And I thought, okay, this is going to get used a little bit. They were images. So you can kind of see where this is going. I'm like, That's not a big deal. It's only three kilobytes. But unexpectedly, the consumers loved the feature that it was supporting.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Yeah, so kind of back to the technical debt, I have an unfortunate story of something that I inherited one time where I think metaprogramming is awesome and can do a lot of really cool things and can really get you out of a bind in certain situations. But then it can also be overly abused. And
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
I was searching for a function that was not working properly within Ruby, and I couldn't find it in the code base at all. So I thought, okay, well, surely that this is in the gem or something. So I started looking at all the gems that's included into this Rails application, started tearing apart the gems, opening them to search for this function. Still couldn't find it.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Turns out they were doing a class eval on something that's pulled from the database. So they actually stored Ruby functions as column or data within a column on the database. And that's what was getting executed. That's where the function was defined. So to me, that's a, what's that?
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Yeah, so other than you could not possibly even test that bit of code with any kind of reason, but it was a nightmare.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
So just a warning to when you think that you're doing something really cool and elegant that's avoiding code duplication or whatever, I would much rather have code duplication all across my application than having that level of obfuscation where you're never going to be able to even remotely troubleshoot it.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Yeah, absolutely. And it could even be taken to something like private methods, where if you have a class which has a bunch of methods, start sorting them out, which ones are private methods so they do not need to be accessible to the consumer. Because I've had situations where I've worked on a class that grew over a thousand lines and And there were hundreds of methods in there.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And now that single table is over 30 gigabytes. And it has millions upon millions of records. I'm like, oh, that was unexpected. But I guess that's kind of where I did not think of scale at the time. or proper way. So introducing that kind of technical debt kind of painted us in a corner because now transitioning away from that model is going to be a pain when you're dealing with that much data.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And I had no idea which ones were publicly accessible that were truly supposed to be publicly accessible and which ones were really meant to be private. So not having that level of abstraction, so to speak, you lose a lot of visibility in how important is this class to the consumer.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
So you gave a talk last year about death by 1,000 commits. Could you give us a high-level overview of that talk and kind of some of the things it entails?
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
So Kyle, would you mind telling us a bit about who you are, who you work for, and some of the things that you're doing?
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Yeah, I've had some include statements which spanned 50 lines on some projects I inherited. It's insane, the kind of data that they're trying to return. But yeah, it's crazy. Good advice. Was there anything else? that we want to talk about? I know we're getting at about that time.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Awesome. Well, I'm going to move us over to some picks. Luke, do you want to start us off?
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
I must say that's a great pick. All right. Hey, Matt, you want to chime in with some picks?
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Right. And so would you mind also explaining what technical debt is? What would you consider technical debt? And what are some things that you would maybe not consider technical debt? Kind of like some debusting myths about technical debt.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Awesome. I'll jump in with a couple of picks. One is from Google. It is the Titan Security Key. Other companies have similar products like the UBO Key. It's a USB or a NFC key that will do your authentication for you. So I actually have a couple of these arriving in the mail today in preparation for another Drifter Ruby episode that I want to do on these things.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
So that should be a pretty interesting one. I don't think it's going to have too much popularity because I never had one of these keys before or later today. And the other is I have now in front of me a little rack of Raspberry Pi 8GB of RAMs that I'm building into a tiny Kubernetes cluster for, well... Just because I can, really.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
So I love Raspberry Pis, and they just released their 8GB versions, which actually makes it nicer to run some heftier things on it now. Still slow, but still a lot of fun. All right, Kyle, do you want to join in with some picks?
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
All right. Fair enough. Well, it was great talking to you, Kyle. And I always like talking about technical debt because I am notorious for introducing it.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
All right. Well, that's a wrap for this episode. We appreciate you coming and talking with us. It was a lot of fun.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Mm-hmm. Yeah, I definitely have to agree with that. So what are some of the real world examples that you guys have experienced over your years where at the time you made a decision and you or the team thought like, this was a great choice. This is the right way to do it. But then later you found that it became more troublesome or more of a headache than it was worth.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Hi, everyone. Welcome to another episode of Ruby Rogues. I'm David Kimura. And today on our panel, we have Matt Smith. Hello. Luke Sutters.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And that could have its own issues if you have thousands of requests per second coming in. So yeah, definitely not a fun problem to solve. And it's also, I guess, good to know what kind of migration or really what kind of SQL functions will cause a table lock. So adding an index or adding a column and stuff can lock your table.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
So being aware of what actually is going to lock the table is really good information to know.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
So that kind of covers the database and schema side of things. To step away from the code, you had mentioned about onboarding people. With a larger client base, what does that process look like for you guys? And how do you really bring a junior or mid-developer into the company and have them productive quickly?
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
Yeah, that's kind of one of my pet peeves, I guess you could say, with linting is that if a particular project has a set of practices it likes to follow, maybe it is no more than 100 characters on a line. That kind of feedback should never happen in a code review. If you have those kind of expectations, then they need to be known expectations via a linter, whether it's RuboCop or standard RB.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
And it should never be an unknown expectation to the developer. So I'm definitely on board with that. And that's something that I've had to fight and struggle with is going through code reviews and having everything kind of nitpicked. Because when it decreases the morale of the developer, if every pull request they're making, it's just getting bombarded with styling quirks or requests to change.
Ruby Rogues
Rails at Super Scale with Kyle d'Oliveira - RUBY 667
So I could definitely... agree with that point. And I think that every project should adopt some kind of linter if there are expectations of what they're doing. Even if you bring in RuboCop, you disable everything by default and then you just start adding in or allowing which exceptions your team follows on that particular project.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
I will say that if you've got relatives who are in the medical profession, especially if they're pathologists, even the use of the term post-mortem makes me uncomfortable because those are no fun at all. But, yeah, it's also a word that we use. So, yeah, it just makes me – oh, it's creepy – It's all zombies. I don't know.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
Yeah, the post-mortem brings me flashbacks to episodes of the X-Files in the 90s when Dana Scully was taking an alien apart.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
I just got a weird brain, all right? It's what my brain thinks of.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
It was me. Just to be clear, was this incident a monitoring problem or an alerting problem? Because it sounds like an alert did go off at some point.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
Yeah, makes sense. I mentioned, Dave, you've been through... like me, many different monitoring platforms, Datadog, you said New Relic, which are the good monitoring platforms? Or which ones are you like, this is the platform that works really well for this API situation?
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
Do you find that putting really large screens on the office wall helps make your application more reliable?
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
The Slack channel truly is the giant performance monitor of 2020. That is literally what tells me whether stuff is working at the moment. I'm thinking there are a lot of people in the same boat. So it sounds like you're saying that once you get to a certain stage... then the off-the-shelf monitoring isn't really going to cut it. So you have written custom monitoring for your application.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
Is that correct?
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
Did you do a background check on me before this episode?
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
Back in 2015, I was working in the States and due to various issues, I was still responsible effectively for a bunch of servers in the UK. And I'd gone to see a film and put my phone on silent. And of course, all the servers melted halfway through Skyfall or whatever movie it was. Tom Cruise did not alert me of the impending server disaster while he was dealing with the aliens.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
So I came out and everyone was very upset. So I ended up writing custom alerting with a custom app using the Android automator that when it received a text message with the magic string in it would actually like turn up volume and then play the Beatles help at full volume. And that worked. That worked very well.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
But what it didn't have, which I like on the patient duty system, is the acknowledgement. So you can see, you know, yeah, I've sent the message. Has that person seen that message? And, you know, tap the yes, I am aware service a melting button.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
Yeah, just the spammers, right? That's a tragic thing to say, Dave.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
Can I ask you about composite monitors? Because that is a phrase I have not heard before. I'm familiar with a rate monitor. My understanding of that is if it drops really quick, it goes off. But if it drops slowly, it doesn't go off. But what is this composite monitor?
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
So you're looking at multiple different things at once. Is that so that you could combine those to kind of set effectively a much lower threshold and get higher signal-to-noise? So you say something like, well, we'll allow this number of 404s or allow this number of server load this number of other errors. But if you get all three at the same time, then it triggers something different.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
Or does it use a lower number? What's the result of that? The advantage of using that logic instead of just saying, here is the minimum number of 404s. Here is the maximum number of 404s. Here's the maximum number of errors. How does that actually translate into a better metric?
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
Cool. Luke, how about you? I just said it's a really interesting. Is this something you wear all the time, Dave?
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
Just kidding. Yeah, just checking it, just obsessing about it. I suppose that's good. It's not real time. Otherwise, that'd be even more stressful because you'd be sitting there and go off and say, yeah. Blood pressure's going up. Get caught in the feedback loop.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
I've been fighting the code this week, Chuck. I've been building strange command line interfaces in Ruby, and I've been using a little application which is installed by default on most Ubuntu-based systems called Whiptail. This is an old-school text-style interface for when you can't put a GUI on it for various reasons. So this is kind of like, it makes it look more professional.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
It makes it look like a real piece of software. And using this from Ruby has been a real pain because you need to do funny things with file descriptors to get the user data out. So it turns out a very nice man by the name of Felix C. Sturgeman has written a gem There's written a gem to do it all for you in native. Way to go, Felix.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
So, yeah, you know, all of that work I did was totally unnecessary. And you, too, can build amazing old-school ASCII-looking interfaces using the gem. It's called EFE, and it's on GitHub under the obfusc. And there's loads of really interesting utilities online. on Neog Faster Kit Hub.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
If you dig in, there's some interesting low-level stuff for when you want to kind of Ruby yourself up on the command line. So well worth a look.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
What was the fallout? Did everyone like phone up and get really angry? Oh, from a customer perspective? Yeah, yeah. This is the best bit of outage stories is the kind of the human cost of whoever has to answer the phone the next week.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
And I guess for this specific, not this specific product, but kind of product where your customers are consuming your API, you're also at the mercy of their implementation too. So, you know, they're making a kind of call against you. And if that call is failing, you know, you've got to hope that their system can cope with that as well.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
Can we talk about the blameless culture for a bit? This is a new idea. And when I was managing engineering teams, I used to have what I called the finger of blame. So I used to do it the other way around. I would hold up my finger in a meeting and I'd introduce the finger as the finger of blame. And then we'd work out who the finger of blame should be pointing to.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
Now, more often than not, of course, it was me. So the finger of blame was a double edged finger. But it was it was a kind of way of, you know, people take it very seriously when they mess up the kind of stuff. So you kind of have to get your. get your team back on board. So it was a way of kind of lightening the mood after that week's disaster.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
But a blameless culture, as you said, is the kind of more sophisticated way of doing it instead of pointing a jovial finger at the person who messed up. What does that look like? I mean, you know, do you just go around telling people it's not their fault? Or, you know, how do you implement a blameless culture in what sounds like quite a big engineering team?