Paul Zaich
👤 PersonAppearances Over Time
Podcast Appearances
I think it starts for us with, it really started with our CTO, Jonathan, and co-founder, making that a priority from pretty much day one, basically from the beginning of our process when we've had issues or incidents. We've done a postmortem doc. We've had a process around that. And it's always been very forward-looking. facing very, very much about what could we have done better?
I think it starts for us with, it really started with our CTO, Jonathan, and co-founder, making that a priority from pretty much day one, basically from the beginning of our process when we've had issues or incidents. We've done a postmortem doc. We've had a process around that. And it's always been very forward-looking. facing very, very much about what could we have done better?
What can we improve? What are the things we should be doing going forward? So I think having that first touch point and really having that emphasis from the beginning was really important and cascades down. I think as you're building out a bigger engineering team, that's critical as be able to just continue to build, keep that culture going.
What can we improve? What are the things we should be doing going forward? So I think having that first touch point and really having that emphasis from the beginning was really important and cascades down. I think as you're building out a bigger engineering team, that's critical as be able to just continue to build, keep that culture going.
And I think that's definitely a challenge to continue doing. But I think as we've grown, we've been able to do that so far. So I think that was step number one. I think a second piece of it is understanding and trying to understand when it's more of a process issue versus something that someone particularly did wrong.
And I think that's definitely a challenge to continue doing. But I think as we've grown, we've been able to do that so far. So I think that was step number one. I think a second piece of it is understanding and trying to understand when it's more of a process issue versus something that someone particularly did wrong.
And I think a lot of the time, I think a lot of incidents do occur because you're trying to make different prioritization decisions and you're trying to make sure that you anticipate things in advance or failure, failure moments. And sometimes you just miss those. And those are particular cases where I think the management team needs to really take responsibility for it.
And I think a lot of the time, I think a lot of incidents do occur because you're trying to make different prioritization decisions and you're trying to make sure that you anticipate things in advance or failure, failure moments. And sometimes you just miss those. And those are particular cases where I think the management team needs to really take responsibility for it.
It's not an individual issue that caused that particular downtime or that it wasn't that necessarily that one piece of code. And so it could be just an example. I mean, this is an example, I think, actually, where we had some technical debt, we were trying to clean it up. And that was a good thing.
It's not an individual issue that caused that particular downtime or that it wasn't that necessarily that one piece of code. And so it could be just an example. I mean, this is an example, I think, actually, where we had some technical debt, we were trying to clean it up. And that was a good thing.
But I think we didn't necessarily have everything in place to be able to address that technical debt effectively. And that's not necessarily one engineer's responsibility to be out in front of.
But I think we didn't necessarily have everything in place to be able to address that technical debt effectively. And that's not necessarily one engineer's responsibility to be out in front of.
Absolutely. And I think one other thing to highlight here is that when you don't have a famous culture, folks are going to be very afraid to speak out when they do see an issue, whether it was there, they think it was their mistake or someone else's. They're not going to want to escalate that issue and make sure that it gets attention necessarily.
Absolutely. And I think one other thing to highlight here is that when you don't have a famous culture, folks are going to be very afraid to speak out when they do see an issue, whether it was there, they think it was their mistake or someone else's. They're not going to want to escalate that issue and make sure that it gets attention necessarily.
And so one of the best side effects of having a blameless culture is that you get really engaged response and everyone's going to work together to try to address the issue. I think that even cascades down to customer communication as well, because when you're really engaged in trying to do that, then you're doing the best thing possible. for the customers as well.
And so one of the best side effects of having a blameless culture is that you get really engaged response and everyone's going to work together to try to address the issue. I think that even cascades down to customer communication as well, because when you're really engaged in trying to do that, then you're doing the best thing possible. for the customers as well.
These are trying to address these issues head on, not try to find ways to kind of smooth them out under the surface.
These are trying to address these issues head on, not try to find ways to kind of smooth them out under the surface.
You are. I think at the end of the day, you're going to try to find the root cause, right? You're going to look for that commit. You're going to look for the log. Maybe it was a script that was logged into your logging system, whatever it is. You're going to look for that and look for the root.
You are. I think at the end of the day, you're going to try to find the root cause, right? You're going to look for that commit. You're going to look for the log. Maybe it was a script that was logged into your logging system, whatever it is. You're going to look for that and look for the root.