Dave Kimura
Appearances
Ruby Rogues
Developing your development - RUBY 649
Right. That's pretty cool to be able to block out your colleagues with a click. If only I had that work today. Is there an API?
Ruby Rogues
Developing your development - RUBY 649
So this is operating off a Visual Studio hook, is it?
Ruby Rogues
Developing your development - RUBY 649
No, keep it out. That would be my excuse. I'd be like, why aren't you doing any work? Oh, it's in BIM. It doesn't work in BIM.
Ruby Rogues
Developing your development - RUBY 649
I like to be demoted. Dave has just made his background of his room go green and red in synchronicity with what he was saying. That was very impressive. Yeah. That's automation.
Ruby Rogues
Developing your development - RUBY 649
fluctuating it up and down is probably distracting but just going in stress level yeah like it's red it's red i'm not doing good oh no get it back to green yeah that's that doesn't actually help with coping do your lights go red when your tests go red they what tests
Ruby Rogues
Developing your development - RUBY 649
I think I'd rather be Hugh Jackman, really, wouldn't you? Dude at 95. I think I'd rather be there. This is the image I have of myself. 3 a.m., cigarette in one hand, mixed drink in the other. Maybe two keyboards and a laptop top left. I mean, crikey. Now that I've learned that sensible office hours are the key to programming greatness, I'm kind of reconsidering my career, really.
Ruby Rogues
Developing your development - RUBY 649
I'm going to become rich and famous, rich and more famous, after I create a YouTube plugin for VS Code that enters random keystrokes. There you go. Okay.
Ruby Rogues
Developing your development - RUBY 649
Dude, I've been trying to use the Rubyist app on the iPhone. Do you have this app, the Rubyist app? I don't know. So this is, oh man, I've forgotten who wrote it. It's one of the, pretty sure it's one of the Japanese guys that everyone knows. Anyway, this is Ruby on your iPhone with API hooks into things like Siri.
Ruby Rogues
Developing your development - RUBY 649
So you can kind of automate stuff from your iPhone and you can also kind of write Ruby on the iPhone, right? It's using the mRuby interpreter. So that's how they've got it in there. They've just kind of linked the whole of mRuby into the app. And I'm getting absolutely nowhere with it. I have not got it to do any. And it's me. It's not the app. It's definitely me.
Ruby Rogues
Developing your development - RUBY 649
But it's a fantastic thing to play with when you're waiting for stuff to happen. So that's my pick for this week. It is the Rubyist app. It is free on the app store. Awesome.
Ruby Rogues
Developing your development - RUBY 649
I got a question for you here. So as I'm understanding it, at software.com, you've got loads of really cool tools that can kind of look at people's commits or some other metric you're looking at to assess productivity.
Ruby Rogues
Developing your development - RUBY 649
You've done all the work in that one commit at the start.
Ruby Rogues
Developing your development - RUBY 649
I went on Code Climate the other day. It wasn't my idea. My colleagues did. And it gave my code a D. And quite honestly, that kind of put me off all of these kind of code assessment tools. I'm now a bit scared that if people go to software.com and show them my work, they'll say that, you know, Luke's only productive on like a Tuesday morning. And even then he doesn't do much work.
Ruby Rogues
Developing your development - RUBY 649
It sounds like a great tool for managers. Is it going to help the software grunts, the frontline coders like me?
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
I will say that if you've got relatives who are in the medical profession, especially if they're pathologists, even the use of the term post-mortem makes me uncomfortable because those are no fun at all. But, yeah, it's also a word that we use. So, yeah, it just makes me – oh, it's creepy – It's all zombies. I don't know.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
Yeah, the post-mortem brings me flashbacks to episodes of the X-Files in the 90s when Dana Scully was taking an alien apart.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
I just got a weird brain, all right? It's what my brain thinks of.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
It was me. Just to be clear, was this incident a monitoring problem or an alerting problem? Because it sounds like an alert did go off at some point.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
Yeah, makes sense. I mentioned, Dave, you've been through... like me, many different monitoring platforms, Datadog, you said New Relic, which are the good monitoring platforms? Or which ones are you like, this is the platform that works really well for this API situation?
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
Do you find that putting really large screens on the office wall helps make your application more reliable?
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
The Slack channel truly is the giant performance monitor of 2020. That is literally what tells me whether stuff is working at the moment. I'm thinking there are a lot of people in the same boat. So it sounds like you're saying that once you get to a certain stage... then the off-the-shelf monitoring isn't really going to cut it. So you have written custom monitoring for your application.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
Is that correct?
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
Did you do a background check on me before this episode?
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
Back in 2015, I was working in the States and due to various issues, I was still responsible effectively for a bunch of servers in the UK. And I'd gone to see a film and put my phone on silent. And of course, all the servers melted halfway through Skyfall or whatever movie it was. Tom Cruise did not alert me of the impending server disaster while he was dealing with the aliens.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
So I came out and everyone was very upset. So I ended up writing custom alerting with a custom app using the Android automator that when it received a text message with the magic string in it would actually like turn up volume and then play the Beatles help at full volume. And that worked. That worked very well.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
But what it didn't have, which I like on the patient duty system, is the acknowledgement. So you can see, you know, yeah, I've sent the message. Has that person seen that message? And, you know, tap the yes, I am aware service a melting button.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
Yeah, just the spammers, right? That's a tragic thing to say, Dave.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
Can I ask you about composite monitors? Because that is a phrase I have not heard before. I'm familiar with a rate monitor. My understanding of that is if it drops really quick, it goes off. But if it drops slowly, it doesn't go off. But what is this composite monitor?
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
So you're looking at multiple different things at once. Is that so that you could combine those to kind of set effectively a much lower threshold and get higher signal-to-noise? So you say something like, well, we'll allow this number of 404s or allow this number of server load this number of other errors. But if you get all three at the same time, then it triggers something different.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
Or does it use a lower number? What's the result of that? The advantage of using that logic instead of just saying, here is the minimum number of 404s. Here is the maximum number of 404s. Here's the maximum number of errors. How does that actually translate into a better metric?
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
Cool. Luke, how about you? I just said it's a really interesting. Is this something you wear all the time, Dave?
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
Just kidding. Yeah, just checking it, just obsessing about it. I suppose that's good. It's not real time. Otherwise, that'd be even more stressful because you'd be sitting there and go off and say, yeah. Blood pressure's going up. Get caught in the feedback loop.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
I've been fighting the code this week, Chuck. I've been building strange command line interfaces in Ruby, and I've been using a little application which is installed by default on most Ubuntu-based systems called Whiptail. This is an old-school text-style interface for when you can't put a GUI on it for various reasons. So this is kind of like, it makes it look more professional.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
It makes it look like a real piece of software. And using this from Ruby has been a real pain because you need to do funny things with file descriptors to get the user data out. So it turns out a very nice man by the name of Felix C. Sturgeman has written a gem There's written a gem to do it all for you in native. Way to go, Felix.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
So, yeah, you know, all of that work I did was totally unnecessary. And you, too, can build amazing old-school ASCII-looking interfaces using the gem. It's called EFE, and it's on GitHub under the obfusc. And there's loads of really interesting utilities online. on Neog Faster Kit Hub.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
If you dig in, there's some interesting low-level stuff for when you want to kind of Ruby yourself up on the command line. So well worth a look.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
What was the fallout? Did everyone like phone up and get really angry? Oh, from a customer perspective? Yeah, yeah. This is the best bit of outage stories is the kind of the human cost of whoever has to answer the phone the next week.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
And I guess for this specific, not this specific product, but kind of product where your customers are consuming your API, you're also at the mercy of their implementation too. So, you know, they're making a kind of call against you. And if that call is failing, you know, you've got to hope that their system can cope with that as well.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
Can we talk about the blameless culture for a bit? This is a new idea. And when I was managing engineering teams, I used to have what I called the finger of blame. So I used to do it the other way around. I would hold up my finger in a meeting and I'd introduce the finger as the finger of blame. And then we'd work out who the finger of blame should be pointing to.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
Now, more often than not, of course, it was me. So the finger of blame was a double edged finger. But it was it was a kind of way of, you know, people take it very seriously when they mess up the kind of stuff. So you kind of have to get your. get your team back on board. So it was a way of kind of lightening the mood after that week's disaster.
Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652
But a blameless culture, as you said, is the kind of more sophisticated way of doing it instead of pointing a jovial finger at the person who messed up. What does that look like? I mean, you know, do you just go around telling people it's not their fault? Or, you know, how do you implement a blameless culture in what sounds like quite a big engineering team?