Jesse Spivak
Appearances
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Again, this isn't a knock against ACA. I think this is just that in our company, we have a ton of infrastructure set up to support the tools that our company has sort of blessed that are kind of frequently in use. In fact, we have like a name for that. We call it the paved road, which is like if you're an engineer, you have a lot of autonomy about what you use.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
But if you stay on the paved road, it should be an easy path. And Akka, which we use with Java or Scala typically, is not on Ibotta's paved road. So we had to kind of have a contentious meeting, a contentious conversation with the architects at Ibotta and say, no, we believe in our developer and we think that she knows what she's doing and we're ready to sort of make our bed and lay in it.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
And hopefully they're mistakes that I will learn from and not make again as I continue to lead projects that I've got in the future. And my hope is that by Sort of articulating these mistakes and what I learned from them, other folks can benefit. And so the four mistakes that I made are, first, we picked the wrong technology. We can get more into that. We also, as a team, we siloed work.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
And they were like, well, you know, as long as you know that. So they let us kind of, they gave us just enough rope to, I forgot how the rest of that goes, to hang ourselves with.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Yeah. So this, and this kind of gets into the next problem, which was the siloed work. So we're starting to work on building this Akka-powered race car of a microservice to pull billing logic out of our Rails monolith. And there's sort of two streams of work. There's the development of the Akka microservice, which is, we were writing, we were using Kotlin, which is a JVM.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
It's kind of a nicer Java. It's what Android apps are written in. It's actually pretty nice. So we have that work going on. And then we have to integrate that microservice, sorry, into the Rails monolith. So I ended up taking on a lot of the integration work and the other developer took on most of the Akka and Kotlin work. So we have these sort of two very isolated pieces of work that are siloed.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
And then something terrible happened. And I don't blame this person at all because I wouldn't want to be on my team Anyway, she decided that she was much more interested in data engineering, and she moved to a different team in the company. And that's actually a great thing about working at Ibotta.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
We've got tons of really cool problems, different spaces, and people are almost encouraged to move around to find things that they like and are good at. So she made this move, and now the problem of using a framework that none of us had experience with and that the general company didn't have a ton of experience with really became self-evident.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Because now there was still work left to do on the microservice. And I'm just a Rails developer. I had to go into there and start writing Kotlin, start reading the ACA documentation and try to wrap my head around what this whole actor system meant. And it was tough. And that's why I call it a big mistake. Sorry, Dave.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Absolutely, Dave. And we were originally in the monolith, the way the system worked. was it ran on a background scheduled job using rescue and the scheduled job basically took roughly 45 minutes to run. So it was running like kind of like a waterfall, like every 10 minutes we'd start a new job that would take 45 minutes to run.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
And so going from that to basically completely real time is a huge improvement. And if real time for 500,000 events per day versus 5 million events per day are different things. But if you're at real time, it's already this just enormous improvement over what we were working with, which is like this 45 minute loop.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
So at that point, and I guess I'll say another thing before I get into how we fixed my mistake. This is actually getting to what Dave said. This is a case of premature optimization. We sort of didn't do the back of the envelope math
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
well enough or didn't have a clear picture of like okay this is really like designed to handle like literally a thousand times more traffic than our best day so you know we we didn't that was that was definitely a mistake like we should have asked that question and realized okay maybe Maybe this is a little too much horsepower. We don't need this.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
So work was divided up in not the best way. We fell into the premature optimization trap. And then maybe worst of all, we made way too many changes at one time. So I can go into detail on any of those.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
It's too much trouble for what it's, for what it's buying us. And on my end also, I was imagining, you know, getting all these events we call it. So the microservice was producing like prediction events. Okay. So predicting every time a piece of content should come down and making an adjustment about when it thinks that that should happen.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
And so I'm imagining the monolith is listening to these prediction events, subscribe to an SNS topic or SQSQ. And I imagine thousands and thousands and thousands of events every second, which is way, way too much. And so I was thinking of all these clever ways to do caching and to try to figure out when can I drop events so I don't need to hit the database?
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
How can I come up with these clever ways to only make round trips to the database when I really need to?
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
And that added time to the development, and it also added a ton of complexity so that when our two systems, when we were like, okay, let's turn them on, let's see what happens, the first error that comes up, because of course there's going to be an error when you first try it out, that was really hard to debug. It was really hard to understand, like, is this a caching issue?
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Is this actually an issue with the data coming from the microservice? Like, it was really hard to isolate that. And then that obviously moves into the third problem, the third mistake was that we made too many changes at one time. So all these were ways that I tried to shoot myself in my own foot while working on this very important project.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Yeah, Luke, that's a really astute observation. So we were changing structure and we were changing behavior at the same time, which is something since that point, I've really tried to avoid doing, even at like a micro PR level. If I have a PR that I'm going to push up, it's either going to add behavior or change behavior, or it's going to change the structure of the code.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
And I'm going to try to not do the two things at the same time. So in this case, we did the two things at the same time. And at multiple points, we should have known or in retrospect, we should have known to pause and verify, right? And if we couldn't verify, take that failure as feedback and iterate. And I think that that's really what really
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
strong engineers, experienced engineers know to do, make one change at a time and verify. I guess it's kind of like when I run RSpec, right? Or RSpec feels like supernatural to me. So every time I hit my test suite, I'm like, okay, I expect this to happen. And like, sometimes I'm right. And then sometimes I'm wrong. And that's like really interesting too.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
But making that initial guess of what's going to happen is super important. And I think we need to slow down and stop and say, okay, here's our guess as to what's going to happen. Here's how we're going to verify it. And if we can't verify it, here's like the corrective action we can take.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Yeah, so what did we do about it? Unfortunately, well, not unfortunately. It was a good learning experience. But obviously, my happy place is deep in our legacy Rails application environment. finding all the pathways there, but I had to get really pushed myself, get out of my comfort zone. I had to pick up Kotlin.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Luckily, Kotlin is like a super friendly language, I think to get into, especially for folks who are coming from Ruby. Cause I think that there's like kind of like an emphasis on syntax on like syntactic sugar on like making the code actually like readable and look nice. Whereas that's not always the case with all JVM languages. So in that case, in that sense, it was kind of friendly.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Also at Ibotta, we were able to organize kind of a learning group. So there were lots of folks who were new to Ibotta or new to Kotlin who were kind of trying to solve these similar problems. So we made a study group. We found students. cool online coursework. We held each other accountable for making sure that we were making progress on those things.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
And I started writing Conlon in the microservice to try to, like you said, push across the line. We need to get this across the line and then we can kind of circle back and deal with some of the underlying problems.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
And that's important because as engineers, at the end of the day, we're trying to deliver value to the businesses that we work for and not just trying out new technology or optimize what technological solution we're implementing.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
So we have, we have problems with it and we decided let's get this thing. We're like, we're close enough, even with the problems, even with our lack of understanding, we're close enough that a total rewrite right now would put us back a long way and upset the business. So let's push it across the line. And then we can circle back and figure out what to do.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
So delivering that value, even though it kind of felt yucky, even though we knew that there were things that were going to need to change over the long term, bought us some time and bought us some credibility in the organization.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
I think the most experienced engineers, one of the best engineers I've worked with, his name is Justin Hart. He did the first commit on the monolith that I work on. And every time I work with him, that is always my experience. He's like, we're making this change. Here's what we expect. And we're going to do it. and then we're going to verify it, and then we're going to move on.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
And I'm always like, oh, why don't we do this, this, and this? And he's like, no, no, no, just this one thing. And it's illuminating, right? It's illuminating. I think it's helped me as a developer in the projects that I'm doing now, realizing like, okay,
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
This thing that I want to do, it's actually like four different things, and I'm going to do the first thing first, verify it, and then move on from there.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
And absolutely. And I just want to kind of call out that there's like a certain amount of privilege that comes with being able to talk about our mistakes, right? I'm not worried that my boss is going to fire me. And I'm also not worried that Folks won't take me seriously for giving this talk. If anything, it probably improves my reputation as evidence by getting to talk to three of you gentlemen.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Yeah, I mean, I don't know. Have you all checked out the... Basecamp's email, hey.com.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Yeah, so I've been using it, and obviously I'm a Basecamp fanboy, but I think it's pretty amazing what they're able to do with just HTML over the wire for the most part and not relying on some of the frameworks that have come out recently. Also, I think it's super interesting, like the 2020 Ruby on Rails community survey. If you look, the one put out by Planet Argonne,
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
If you look at what JavaScript libraries people are using, so I was expecting that number one would be React. Maybe number two would be, I don't know, Vue or Ember maybe. But number one on there is jQuery. And it's like we're all in this jQuery world. And it's not bad. I love jQuery.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
So I just want to call that out because I think it's important.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
I almost feel like it's, I go into like meditative state almost. It's, it's, it's kind of like the flow state kind of. And yeah, There are times where it's like, I pick up a story and say, okay, yeah, this is easy. I can just kind of like half think about it and do it.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
And when that doesn't work, which it almost never does, I have to like get into this like very focused, I'm making this one change verifying state. It's almost like the cartoon Avatar going into your Avatar state, if you guys know that amazing cartoon.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Avatar and the Last Airbender. I think it came out like 10 years ago, 15.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
He's got an arrow.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
it getting into that really focused state, actually that costs something, right? It takes like a little bit out. You can't maintain that state forever. There's like a mana pool almost of how long you, how long for me I can be in that extreme focused state.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
And I think that when there are times where I look at a problem and I'm like, do it like, I almost like ask myself, do I need to be in this highly focused state to get this thing accomplished? And more often than not, the answer is yes, but my initial answer is no. And so I end up wasting time by not doing things as systematically as they need to get done. Am I alone on this?
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Nobody else goes into Avatar State when they code?
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Yeah, I love that idea of a naive implementation. And I think the Akka microservice was our naive implementation. We didn't consider the scale at all. We didn't consider the makeup of the team changing.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
And as a result, when we went back to fix the naive implementation, when we wanted to get rid of the double loops or whatever, we ended up moving to Java, so from Kotlin to Java and from Akka to Camel. And the reason that those tools made sense is because they were very common in our company. And we didn't have to reinvent the wheel.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
We weren't seeing errors for the first time in the entire company. It was sort of a knowledge base to draw from. So the current state is that we have Java Camel Spring microservice talking to the Rails monolith. It's very stable. It's performing super well now.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
And, yeah, I mean, getting... I guess a question I have is like, how can we speed up the process of getting from our naive solution to the more learned solution?
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
At the point that we were like, you know, we can just do this in Java and Camel, we had wrapped our heads around the problem enough that we really felt like it wasn't hard anymore. And maybe that's what it takes.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Those are the proudest days.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
That's interesting. The person who taught me how to code... one of the first pieces of advice that he gave me was that you should write a lot of code and throw a lot of code out. And that's how you'll get better at coding.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Yeah, absolutely. So my name is Justin Spivak. I'm a senior engineer at a company called Ibotta, which is a cashback for shopping app based in Denver, Colorado. I've been working there for about three and a half years. We are doing some hiring, so check out our careers page. I guess I'm famous, as it were, because I gave a talk at the first remote RailsConf this past May.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Thankfully. Yeah, so we did a couple of things I think that helped us. So the first is, as John said, we were open about these things. We didn't try to hide that things weren't going as well as we had wanted them to. And I think that Ibotta has a pretty strong culture in the sense that We're not trying to throw people under the bus in engineering.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
If something crashes, it's not who's going to get fired. It's like, okay, how do we learn from this? This is a mistake that cost us some money. How do we make sure that that money is actually teaching us something? So that was part of it. And then I think also we did a good job of communicating to external stakeholders.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
We communicated to the finance team who were kind of one of the main consumers of the data that we were producing. and really went through in detail. Here's where we're at. Here's the timeline, like updating them. We were checking in with them all the time and just keeping expectations in line, I think really helped us out.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
So even though we delivered a little bit later than I think we thought we would at the onset of the project, because we were able to communicate that, we were not fired. And yeah, I mean, not only that, We've hired more people. We're still hiring.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
And if you're thinking about getting into the mobile coupon space, there's a ton of really cool problems, even if you're not passionate about mobile coupons. And you might get to talk to me in the interview process, which will be fun.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
John, yeah, I agree with what you're saying. I think that for a lot of us, talking about mistakes openly and honestly, and what we learn from them actually builds our credibility. But that's not always the case, depending on sort of how you present what you look like. I think that, you know, you guys recently did an episode, I think, where you talked about issues with
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Yeah, so maybe everybody has experience or a lot of folks have an experience in, let's just say, a not-mean-us-one interview, an interview that maybe is a little more hostile than we'd like.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
And I think a lot of us have experienced kind of broken interviews where it feels more like the person on the other side of the table is trying to prove how much smarter they are or how much better they are at coding than I am. And that's not nice.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Another thing that's not nice is asking someone to do an inordinate amount of work outside of work that's not paid in the form of like a take-home project. So I've done take-home projects that have taken me an entire weekend, multiple days, and that's uncompensated work. And that can bias your process against people who have outside of work commitments like families. And I'm just,
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
you know, who don't want to be working all the time. So I thought it would make sense to kind of take the best part of the Ruby community, this idea that Matt's is nice. And so we are nice and apply it to interviewing. Let's like actually be nice to the people that we potentially could be working with. And, you know, the pandemic has been terrible in so many ways.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
But it did offer us this opportunity to kind of dramatically rethink what our interview was going to look like. Because we're not coming into the office. Everything has to be remote. And basically, our HR team and our leadership were like, how can we do this? We've only been accustomed to bringing people in, asking them super tricky things that they have to whiteboard.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
How can we translate this to a remote interview? And this is what I proposed. And this is like... what we landed on, which is an interview not meant to trick the interviewee. It's an interview meant to simulate what the first couple days of work is going to look like. And it's supposed to give us as an organization a sense of how much we would enjoy
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
this person as a colleague, how successful they'll be. And the message that we're always trying to send is not, hey, I'm so much smarter than you because I understand recursion or because I understand how to do whatever this type of model, this type of data structure.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
The message is you're going to be successful on our team and you're going to like working with us and we're going to like working with you. So the way that we do that is basically by giving the person a sample of code from our domain And it's highly simplified. And we ask them to just read the code. And we say, what is this doing? What do you like about this code?
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
issues of hiring and getting equality in the workplace. And I think that that plays in here for sure.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
What do you not like about this code? And it's not a bug-finding adventure. We're not asking them to find where an error is going to be secretly raised or why a test is failing. You can't really do that in a 20- or 30-minute conversation. We want to hear how this person would approach an alien code base, which is what their first task is going to be on the job.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Then we present them with some data, some award events from our system, and we ask them to manipulate that data with a pretty simple algorithm. We even tell them what the algorithm is, and we ask them to code it. And we say specifically, We don't care about the answer here. The answer is not interesting. We just told you what the answer is. We actually want to see what it looks like when you code.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Now, what's your approach? Are you systematic? Are you making guesses about what should happen and checking yourself? Those are the things that we're looking for and we're not looking for some to see. Do you know this random algorithm from your from your computer science education that you'll never use as a web developer at Ibana. So those are the big pieces.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
And I'm stoked because I got invited to talk at RubyConf, which is coming up in November, where I'm gonna be kind of outlining this. You all got a preview of the content there, exclusive to Ruby Roads.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
They pay for the homework, though.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Yeah, I mean, so far, obviously hiring has been slower than typical for Ibotta, but we are still hiring. And so far, the feedback from candidates has been really interesting because we'll get unsolicited emails about how much people like the process and how they hope that this is the direction that the industry goes in.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
a lot of folks who are interviewing with us are interviewing at other places as well. And I see this as like a competitive advantage for us, right? Does this buy us like $10,000 in salary or $5,000 in salary or something like that?
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Like, does this make our company more, you know, once you get to a certain point, there's like diminishing returns on all these different levers that a company can offer to a developer. And I think that this is maybe an unexplored lever or a lever where there's a ton of room to gain. And when, Someone goes through our process and comes out feeling like, hey, I'm going to kick butt there.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
The people there are super nice. They didn't make me feel like an idiot. And they contrast that with other interview experiences that they've recently had. Hopefully that will play in our favor.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
I love that image of shoving a framework under someone's fingernails, Luke. But the thing that I noticed, and I've done a lot of interviews that I've done, I would say easily 200 interviews, over the past two years. We do a ton of hiring, so it's crazy.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
And the thing that I noticed is that when we were asking people to whiteboard, to answer these tougher algorithm questions, I didn't always see a one-to-one correlation between the people who crush that kind of question and the people who I really enjoy working with in either direction. There are false positives and false negatives.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
And I just found there to be way too much noise in that type of diagnostic tool for me to really make a strong decision that I felt confident with that proved out over time. And I'm hoping that this is a remedy for that.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Yeah, there was a RailsConf talk last year. I believe the guy's name is Eric. I'll throw it in the chat in a second. He's from TestDouble. And he talked about, TestDouble is like a Ruby consultancy based out of Columbus, I believe. And he talked about their hiring process
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
And more in the sense of our goal is to build a process that lets candidates show off their strengths and as opposed to helping us find their weaknesses. And that talk really informed our process at ibotta. We're trying to find people's strengths and trying to figure out where they're going to be able to make the most impact in our company.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
I tweet from PlanetEfficacy. So if you can spell that, you can find me. And you'll find a lot of interesting Ruby content, political outrage, and Vermont progressive rock. on that Twitter stream.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
So the high level, at Ibotta, we have just a wonderful, majestic monolith Rails application. Actually, originally, the application was written in Scala. And then after about a month of that, they switched it over and rebuilt the whole thing in Rails. And it's been Rails ever since. So it's going on close, almost 10 years at this point. So it's a large application. It serves millions of users.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Yeah, I've got a couple of picks, different categories. So first is a talk that I watched recently that maybe you all have seen already, but I recommend revisiting it. So Sammy Metz's RailsConf 2019 keynote, which is called Lucky You. And I think it's especially timely as we approach election season. And thinking about equality and issues of inequality in our country.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
So I highly recommend that talk. It's amazing as all Sandy Metz talks tend to be. Then I have, when the pandemic started, we started working remote. I made a small investment in my work from home setup that I highly, highly, highly recommend. I went out and I got an ErgoDox split keyboard, having some wrist pain on the keyboard I was using.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
So I know keyboards can be a whole other podcast, but check out ErgoDox. They make a really, really, really good product that's worth the price. And I've been a huge fan of it since getting it. And then my final pick, there's a book that just came out. It's a guilty pleasure. It's called The Trouble with Peace by Joe Abercrombie, a British gentleman.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
And it is in the grimdark genre of fantasy literature. And I highly recommend it.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
I listened, I, a long time listener, first time guest, and this was, this was awesome. I appreciate it.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Very cool. And about three years ago when I joined, we started thinking, we started growing the team and thinking about how we could like many people have, pull pieces out of the monolith into microservices. So this project in particular was about taking a piece of billing logic from the system, from the monolith, and pulling it out into a microservice.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
The hope was to make it better encapsulated, easier to iterate on, isolate dependencies, every reason that you'd think to build a microservice. So we chose, actually, before I say the technology, before I get trolled, by all the lovers of this technology. I'm going to preface this by saying I don't think that this technology is wrong and I don't think it's bad in and of itself.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
I just think it was not the right technology for the problem and the team.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Absolutely, whoever uses this technology is definitely making a big mistake. Spoiler, we weren't building a side Rails app microservice, which probably would not have been as big of a mistake. But the issue really was that our team, the small team of developers, and then the larger team of engineers in the company, really did not have a ton of experience with the framework that we chose.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
And as a result, we ended up having to do a lot of plumbing and reinventing the wheel and just not benefiting from the institutional experience that exists at Ibotta. And unfortunately, this could work if you're doing kind of like a proof of concept, like let's show what this technology can do. Let's pick a pretty isolated use case.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
But the billing logic that we were pulling out about Monolith was basically do or die. If it did not work, it costs millions of dollars to fix, or it ends up costing the company millions of dollars. So we were walking on a tightrope, and there was no net underneath us. And unfortunately, we decided to, I guess, walk on our hands instead of go across normally.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
And I talked about kind of how crummy of a developer I am. Yeah.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
So the framework that we use is called ACA. And I think for a team that knows ACA, this probably would have been really a perfect tool for the job. But our team and our company really did not have a ton of experience with ACA. And so unfortunately, we weren't able to sort of take advantage of it and use it in a way that sort of professional ACA developers likely can.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Right. It deals with data streams and passing data along in a functional paradigm. And it's meant to accommodate high volume data across highly parallelized system. So, you know, at the time. We went there. Well, I'll talk about why we went there in a second. But in retrospect, it was something that could handle basically 10,000x really what we needed in terms of what it was designed to handle.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
So just sort of on paper, it probably wasn't the best move in that respect. But I could also talk about the team as well and why it wasn't a great fit.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Sure. Yeah, perfect. So the problem that we're trying to solve, and you have to know a little bit about the Ibotta app. So I assume all of you guys have downloaded it and are actively using it.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
That's funny. We did assert, well, I'll get to that later. But basically, Ibotta is a way for you to get digital coupons. Brands put offers in the app. You click on the offer. You show us evidence that you bought the thing that is on offer. And then Ibotta will pay you cash back. They'll send it to your PayPal account, give you a gift card to Amazon, whatever you want.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
So the problem that we're trying to solve is how do we make sure that the offers in the app don't exceed the budget that is allocated to them by the brands that put those offers in the app? And that sounds maybe like an easy problem, like there's an easy way to just say, okay, there's $500,000 in budget for Oreo coupons.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Just divide $500,000 by how much money we're giving out per coupon, and then you know. But it's actually much easier Like it's obviously much harder than that. And in order to preserve a good user experience, we need to make sure that we're not yanking content and surprising our users. Like you would be really upset if you went to the store specifically to buy Oreos to get coupon.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
And then by the time you checked out, the coupon is no longer in your application. So we have to run some predictive algorithms to basically guess when we're going to run out of money and kind of slow the velocity down as we approach that point.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
We are only in the United States right now. Sorry, Luke.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Yeah, it's interesting that you say that because right when I joined the company, I was kind of put on...
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
the team I was actually giving you my my tech buddy my mentor when I joined the company this was his project and as someone new in the company it was it was very it was really overwhelming problem space because basically these campaigns and our application almost have like a physical momentum to them so if you imagine trying to stop a moving train you can't just
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
And, and there, there's no substitute for actually watching this fantastic talk. But more seriously, the talk is really about my experience as a tech lead at Ibotta working on a pretty critical project over the course of about six months or so. And over that time, I made four very big mistakes that put the project in jeopardy.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
hit the brakes and expect it to stop on a dime. You have to apply the brakes over some distance to slow the train down. And that's really how the content in the application is modeled. And it's, I'm not a physics person. And so it's really confusing.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Yeah, that's sort of right. So we have an event-based architecture where our system, and for folks who aren't familiar with that, that means that basically your system publishes events, which is data, that signify that something has happened of interest in the system. So maybe like shopping cart loaded is an event that you might have in like a typical inventory space or something like that.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
So we have events that we're interested in, like content awarded, which means that John went to the store, submitted a receipt through the app and got cash back. So the content has been awarded. So we listen for those events in order to keep track in real time of how much budget is being used. And we basically track that over time to make a rough prediction about how fast things are moving.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
And so ACA, sorry, John, to get back to your original question, ACA is really good at streaming data, at streaming large amounts, high volume data. And Ibotta will get on the order of several hundred thousand of these content awarded events per day, which seems like a lot, but it's actually much lower than I think what ACA can kind of deal with or is designed to deal with out of the box.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
Yeah, perfect. So, yeah, we basically... This comes down to some team issues again and not an issue with Akka. So the team issue was basically that at the start of this project, we scaled up our team. We're like, this is a lot of work. We need to bring in some artillery. And we brought in a new developer from outside the company who is awesome. She's a rock star.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
And it was coming from the ad product space. And with dealing with volume of streaming data at a scale much higher than what we needed or we're going to be dealing with in any near future.
Ruby Rogues
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
and you know she was coming from i believe an aka shop and so she joins the team we're excited to have her we think she's a rock star i mean she is a rock star and she's like this is a perfect use case for aka we're like okay never heard of that but you know we trust you you crushed our interview and we think you're amazing so yeah let's that sounds pretty good and