In this episode of Top End Devs, host David Camira is joined by panelists Luke Stutters and John Epperson, along with special guest Jesse Spivak, a senior engineer at Ibotta. Jesse shares his experiences and insights from a challenging project at Ibotta, where he navigated through four critical mistakes. These included choosing the wrong technology, siloing work, falling into premature optimization, and making too many changes at once. Jesse explains how these mistakes jeopardized the project but ultimately led to valuable learning experiences. The conversation also touches on the importance of discussing and learning from mistakes openly, the complexities of transitioning to new technologies, and the significance of making systematic, verified changes. Additionally, they delve into the evolving landscape of developer interviews, aiming to create a more inclusive and positive experience. Join us as we explore the trials, lessons, and growth that come from navigating the highs and lows of software development.Become a supporter of this podcast: https://www.spreaker.com/podcast/ruby-rogues--6102073/support.
Hey, everyone. Welcome to another episode of Ruby Rogues. I'm David Kumira. And today on our panel, we have Luke Sutters. Hi. And we have John Epperson.
Hello, everybody.
And today we have a special guest, Jesse Spivak. Great to be here. So, Jesse, would you mind explaining just a bit about who you are, some of the things that you're doing, who you work for and why you're famous and all that good stuff? Sure.
Yeah, absolutely. So my name is Justin Spivak. I'm a senior engineer at a company called Ibotta, which is a cashback for shopping app based in Denver, Colorado. I've been working there for about three and a half years. We are doing some hiring, so check out our careers page. I guess I'm famous, as it were, because I gave a talk at the first remote RailsConf this past May.
And I talked about kind of how crummy of a developer I am. Yeah.
I think we can all relate to that on a daily basis sometimes. So would you mind giving a bit of a, you know, highlight talk about what you covered at the conference and stuff, so we can just kind of pick it up from there. We'll link to the, in the show notes, a link to the conference, but just for those who maybe didn't see it. Sure.
And, and there, there's no substitute for actually watching this fantastic talk. But more seriously, the talk is really about my experience as a tech lead at Ibotta working on a pretty critical project over the course of about six months or so. And over that time, I made four very big mistakes that put the project in jeopardy.
And hopefully they're mistakes that I will learn from and not make again as I continue to lead projects that I've got in the future. And my hope is that by Sort of articulating these mistakes and what I learned from them, other folks can benefit. And so the four mistakes that I made are, first, we picked the wrong technology. We can get more into that. We also, as a team, we siloed work.
So work was divided up in not the best way. We fell into the premature optimization trap. And then maybe worst of all, we made way too many changes at one time. So I can go into detail on any of those.
Yeah, and I think it's fair to say that I, our listeners, other members on the panel have been there before.
Yeah, I've worked with loads of people who've made those mistakes. Obviously, I've never made them myself, but oh, my words.
Luke is the perfect developer, let's be fair. No, I was actually going to jump in and say that, like, I don't feel like I'm alone, but usually when you make a mistake, it cascades into more. For whatever reason, I generally feel like you either are rolling along and you feel pretty good about stuff, Or you're just like in a bottomless pit. And there's very little in-between time.
You just suddenly notice that you're at the bottom. So yeah, just saying.
And absolutely. And I just want to kind of call out that there's like a certain amount of privilege that comes with being able to talk about our mistakes, right? I'm not worried that my boss is going to fire me. And I'm also not worried that Folks won't take me seriously for giving this talk. If anything, it probably improves my reputation as evidence by getting to talk to three of you gentlemen.
So I just want to call that out because I think it's important.
I think that. OK, so so speaking of that, I think there's a give and take there, too. So I think this is one of the this will get into a thing that I care a lot about this particular subject, because for most of my shoot, I still feel it today or whatever. It doesn't really matter. I still always feel this pressure. to not let people know about the mistakes that I made, right?
Because letting people know about the mistakes that I made makes me more vulnerable to them not wanting me to work for them, right? Like fire me, whatever the bad thing is that it's a boogeyman. It's not real. It's just a pressure, right?
And so I guess what I'm trying to get at is like, I kind of feel like one of the things that I always care a lot about with mistakes is telling people about your mistakes. I actually... have discovered in reality actually makes people respect you more, But it doesn't change the fact that it's like really hard. And I still feel that pressure to this day.
Yeah.
John, yeah, I agree with what you're saying. I think that for a lot of us, talking about mistakes openly and honestly, and what we learn from them actually builds our credibility. But that's not always the case, depending on sort of how you present what you look like. I think that, you know, you guys recently did an episode, I think, where you talked about issues with
issues of hiring and getting equality in the workplace. And I think that that plays in here for sure.
And I would go as far to say that it depends on what the mistake is, what kind of mistake it is. If it is just utter, incomplete negligence, then those aren't really the kind of mistakes I would really want to be forthcoming and outright about, you know, but
Like if I went in and always had a VPN tunnel into my production environment, then we got malware in our local work environment and that transferred over to production and encrypted all of our production data. I don't know if I would really say like, oh yeah, you know, that was a silly mistake of mine.
No, it's like, okay, not only are our customers affected, but now, you know, my job's in jeopardy because I decided to always take shortcuts. But something like, and what I'm really interested in is your technology framework, or you used the wrong technology. Can you explain a bit more about the scenario? Sure, absolutely.
So the high level, at Ibotta, we have just a wonderful, majestic monolith Rails application. Actually, originally, the application was written in Scala. And then after about a month of that, they switched it over and rebuilt the whole thing in Rails. And it's been Rails ever since. So it's going on close, almost 10 years at this point. So it's a large application. It serves millions of users.
Very cool. And about three years ago when I joined, we started thinking, we started growing the team and thinking about how we could like many people have, pull pieces out of the monolith into microservices. So this project in particular was about taking a piece of billing logic from the system, from the monolith, and pulling it out into a microservice.
The hope was to make it better encapsulated, easier to iterate on, isolate dependencies, every reason that you'd think to build a microservice. So we chose, actually, before I say the technology, before I get trolled, by all the lovers of this technology. I'm going to preface this by saying I don't think that this technology is wrong and I don't think it's bad in and of itself.
I just think it was not the right technology for the problem and the team.
Hold on a second. Hold on a second, Jesse, because the slide I'm looking at says big mistakes. It doesn't say small mistakes. It says big mistakes. So let's just keep this in mind before we reveal what this technology is, what a big mistake was.
Absolutely, whoever uses this technology is definitely making a big mistake. Spoiler, we weren't building a side Rails app microservice, which probably would not have been as big of a mistake. But the issue really was that our team, the small team of developers, and then the larger team of engineers in the company, really did not have a ton of experience with the framework that we chose.
And as a result, we ended up having to do a lot of plumbing and reinventing the wheel and just not benefiting from the institutional experience that exists at Ibotta. And unfortunately, this could work if you're doing kind of like a proof of concept, like let's show what this technology can do. Let's pick a pretty isolated use case.
But the billing logic that we were pulling out about Monolith was basically do or die. If it did not work, it costs millions of dollars to fix, or it ends up costing the company millions of dollars. So we were walking on a tightrope, and there was no net underneath us. And unfortunately, we decided to, I guess, walk on our hands instead of go across normally.
So the framework that we use is called ACA. And I think for a team that knows ACA, this probably would have been really a perfect tool for the job. But our team and our company really did not have a ton of experience with ACA. And so unfortunately, we weren't able to sort of take advantage of it and use it in a way that sort of professional ACA developers likely can.
So this is kind of like a functional programming thing?
Right. It deals with data streams and passing data along in a functional paradigm. And it's meant to accommodate high volume data across highly parallelized system. So, you know, at the time. We went there. Well, I'll talk about why we went there in a second. But in retrospect, it was something that could handle basically 10,000x really what we needed in terms of what it was designed to handle.
So just sort of on paper, it probably wasn't the best move in that respect. But I could also talk about the team as well and why it wasn't a great fit.
Can you talk for just a minute about what was the problem that you were trying to solve? What did you need Akka for?
Sure. Yeah, perfect. So the problem that we're trying to solve, and you have to know a little bit about the Ibotta app. So I assume all of you guys have downloaded it and are actively using it.
Okay.
That's funny. We did assert, well, I'll get to that later. But basically, Ibotta is a way for you to get digital coupons. Brands put offers in the app. You click on the offer. You show us evidence that you bought the thing that is on offer. And then Ibotta will pay you cash back. They'll send it to your PayPal account, give you a gift card to Amazon, whatever you want.
So the problem that we're trying to solve is how do we make sure that the offers in the app don't exceed the budget that is allocated to them by the brands that put those offers in the app? And that sounds maybe like an easy problem, like there's an easy way to just say, okay, there's $500,000 in budget for Oreo coupons.
Just divide $500,000 by how much money we're giving out per coupon, and then you know. But it's actually much easier Like it's obviously much harder than that. And in order to preserve a good user experience, we need to make sure that we're not yanking content and surprising our users. Like you would be really upset if you went to the store specifically to buy Oreos to get coupon.
And then by the time you checked out, the coupon is no longer in your application. So we have to run some predictive algorithms to basically guess when we're going to run out of money and kind of slow the velocity down as we approach that point.
Ooh, dear. That's a dangerous recipe. I must add that I don't think about it is available in the UK at the moment. Is it available in Canada?
We are only in the United States right now. Sorry, Luke.
In my defense, this looks like the perfect storm for me because you've both got the kind of strong financial components. So if you get it wrong, you lose money. And if you get it wrong, then people will have wasted their time going to the shop. You know, in Britain, it's so small. I can walk to the shop. You know, sometimes I just open the window and shout at them and tell them what I want.
But, you know, I know when I was living in the States and people maybe drive for two hours to go to Walmart. So this is quite a problem you've got there.
Yeah, it's interesting that you say that because right when I joined the company, I was kind of put on...
the team I was actually giving you my my tech buddy my mentor when I joined the company this was his project and as someone new in the company it was it was very it was really overwhelming problem space because basically these campaigns and our application almost have like a physical momentum to them so if you imagine trying to stop a moving train you can't just
hit the brakes and expect it to stop on a dime. You have to apply the brakes over some distance to slow the train down. And that's really how the content in the application is modeled. And it's, I'm not a physics person. And so it's really confusing.
So, all right. So this seems a slightly, I mean, it's a twist, right? On your classical inventory problem, right? Is what it kind of sounds like. So the way that ACCA came into this is, is what you were taking, like what all these requests from users and trying to decide whether or not you were going to, I guess, give them the coupon or not or something.
Yeah, that's sort of right. So we have an event-based architecture where our system, and for folks who aren't familiar with that, that means that basically your system publishes events, which is data, that signify that something has happened of interest in the system. So maybe like shopping cart loaded is an event that you might have in like a typical inventory space or something like that.
So we have events that we're interested in, like content awarded, which means that John went to the store, submitted a receipt through the app and got cash back. So the content has been awarded. So we listen for those events in order to keep track in real time of how much budget is being used. And we basically track that over time to make a rough prediction about how fast things are moving.
And so ACA, sorry, John, to get back to your original question, ACA is really good at streaming data, at streaming large amounts, high volume data. And Ibotta will get on the order of several hundred thousand of these content awarded events per day, which seems like a lot, but it's actually much lower than I think what ACA can kind of deal with or is designed to deal with out of the box.
Yeah, so did you consider alternatives? Did you reject them for various reasons? And I guess, so obviously, Akka didn't work out for you. So you must have picked something else that you like better. And how did you arrive at that new thing? And what was that, if that makes some sense?
Yeah, perfect. So, yeah, we basically... This comes down to some team issues again and not an issue with Akka. So the team issue was basically that at the start of this project, we scaled up our team. We're like, this is a lot of work. We need to bring in some artillery. And we brought in a new developer from outside the company who is awesome. She's a rock star.
And it was coming from the ad product space. And with dealing with volume of streaming data at a scale much higher than what we needed or we're going to be dealing with in any near future.
and you know she was coming from i believe an aka shop and so she joins the team we're excited to have her we think she's a rock star i mean she is a rock star and she's like this is a perfect use case for aka we're like okay never heard of that but you know we trust you you crushed our interview and we think you're amazing so yeah let's that sounds pretty good and
Again, this isn't a knock against ACA. I think this is just that in our company, we have a ton of infrastructure set up to support the tools that our company has sort of blessed that are kind of frequently in use. In fact, we have like a name for that. We call it the paved road, which is like if you're an engineer, you have a lot of autonomy about what you use.
But if you stay on the paved road, it should be an easy path. And Akka, which we use with Java or Scala typically, is not on Ibotta's paved road. So we had to kind of have a contentious meeting, a contentious conversation with the architects at Ibotta and say, no, we believe in our developer and we think that she knows what she's doing and we're ready to sort of make our bed and lay in it.
And they were like, well, you know, as long as you know that. So they let us kind of, they gave us just enough rope to, I forgot how the rest of that goes, to hang ourselves with.
Okay. So it sounds like you had an advocate and that's kind of how you landed on it. Where did you go after you decided that Akka was the poor choice? Like, what did you end up with?
Yeah. So this, and this kind of gets into the next problem, which was the siloed work. So we're starting to work on building this Akka-powered race car of a microservice to pull billing logic out of our Rails monolith. And there's sort of two streams of work. There's the development of the Akka microservice, which is, we were writing, we were using Kotlin, which is a JVM.
It's kind of a nicer Java. It's what Android apps are written in. It's actually pretty nice. So we have that work going on. And then we have to integrate that microservice, sorry, into the Rails monolith. So I ended up taking on a lot of the integration work and the other developer took on most of the Akka and Kotlin work. So we have these sort of two very isolated pieces of work that are siloed.
And then something terrible happened. And I don't blame this person at all because I wouldn't want to be on my team Anyway, she decided that she was much more interested in data engineering, and she moved to a different team in the company. And that's actually a great thing about working at Ibotta.
We've got tons of really cool problems, different spaces, and people are almost encouraged to move around to find things that they like and are good at. So she made this move, and now the problem of using a framework that none of us had experience with and that the general company didn't have a ton of experience with really became self-evident.
Because now there was still work left to do on the microservice. And I'm just a Rails developer. I had to go into there and start writing Kotlin, start reading the ACA documentation and try to wrap my head around what this whole actor system meant. And it was tough. And that's why I call it a big mistake. Sorry, Dave.
You know, from my experience, too, is that, yeah, Ruby is slow. You know, there's no getting around that when you compare to a compiled language. But holy crap, is it fast, too. You know, I have a production application which processes over 500,000 active jobs every single day. and it does it extremely quick. I don't need it to be faster. I mean, that's plenty fast.
On the current setup that it's on, which is two cores and four gigs of RAM, and we have two servers dedicated to the background job, so two VMs, it's able to handle that load, and we don't have to worry about it crashing or anything like that. So, I mean, that's good enough for us.
You know, I imagine that it would be able to double that workload before we ever ran into any kind of performance issues where we needed to start scaling up.
Absolutely, Dave. And we were originally in the monolith, the way the system worked. was it ran on a background scheduled job using rescue and the scheduled job basically took roughly 45 minutes to run. So it was running like kind of like a waterfall, like every 10 minutes we'd start a new job that would take 45 minutes to run.
And so going from that to basically completely real time is a huge improvement. And if real time for 500,000 events per day versus 5 million events per day are different things. But if you're at real time, it's already this just enormous improvement over what we were working with, which is like this 45 minute loop.
So at that point, and I guess I'll say another thing before I get into how we fixed my mistake. This is actually getting to what Dave said. This is a case of premature optimization. We sort of didn't do the back of the envelope math
well enough or didn't have a clear picture of like okay this is really like designed to handle like literally a thousand times more traffic than our best day so you know we we didn't that was that was definitely a mistake like we should have asked that question and realized okay maybe Maybe this is a little too much horsepower. We don't need this.
It's too much trouble for what it's, for what it's buying us. And on my end also, I was imagining, you know, getting all these events we call it. So the microservice was producing like prediction events. Okay. So predicting every time a piece of content should come down and making an adjustment about when it thinks that that should happen.
And so I'm imagining the monolith is listening to these prediction events, subscribe to an SNS topic or SQSQ. And I imagine thousands and thousands and thousands of events every second, which is way, way too much. And so I was thinking of all these clever ways to do caching and to try to figure out when can I drop events so I don't need to hit the database?
How can I come up with these clever ways to only make round trips to the database when I really need to?
And that added time to the development, and it also added a ton of complexity so that when our two systems, when we were like, okay, let's turn them on, let's see what happens, the first error that comes up, because of course there's going to be an error when you first try it out, that was really hard to debug. It was really hard to understand, like, is this a caching issue?
Is this actually an issue with the data coming from the microservice? Like, it was really hard to isolate that. And then that obviously moves into the third problem, the third mistake was that we made too many changes at one time. So all these were ways that I tried to shoot myself in my own foot while working on this very important project.
Yeah, makes sense. I'm not sure if I understand it, but so you're saying there's kind of two separate things going on here. Firstly, you're moving to a completely new technology. So you're taking a large part of the app out. You're putting it in a microservice. It's not very micro if you're doing, what, 700,000 things a day. That's not a microservice. This is a macro, micro, or medcro.
And the second thing you're doing is you're actually changing the whole interface. So you're saying you're going from – you previously had a rescue batch job And now he's saying, no, no, no, we're not going to do batching. We're going to do it all immediately.
Yeah, Luke, that's a really astute observation. So we were changing structure and we were changing behavior at the same time, which is something since that point, I've really tried to avoid doing, even at like a micro PR level. If I have a PR that I'm going to push up, it's either going to add behavior or change behavior, or it's going to change the structure of the code.
And I'm going to try to not do the two things at the same time. So in this case, we did the two things at the same time. And at multiple points, we should have known or in retrospect, we should have known to pause and verify, right? And if we couldn't verify, take that failure as feedback and iterate. And I think that that's really what really
strong engineers, experienced engineers know to do, make one change at a time and verify. I guess it's kind of like when I run RSpec, right? Or RSpec feels like supernatural to me. So every time I hit my test suite, I'm like, okay, I expect this to happen. And like, sometimes I'm right. And then sometimes I'm wrong. And that's like really interesting too.
But making that initial guess of what's going to happen is super important. And I think we need to slow down and stop and say, okay, here's our guess as to what's going to happen. Here's how we're going to verify it. And if we can't verify it, here's like the corrective action we can take.
Yeah. So I think you're hitting it like one thing that's like really important, right? So one of the things I think that really kind of makes you a strong engineer of sorts, right? It's not necessarily like to avoid mistakes because shoot, for whatever reason, they keep happening to me, right? It's, are you good at cleaning up, right? Are you good at figuring out that something's not right?
Something smells funny, whatever it is. Can you go back and, you know, sweep up your mess, you know, either push it across the finish line because you're close enough or Or say, actually, this is the wrong path. We should actually back up, go to the last intersection and go down the other way.
So, I mean, it's definitely, yeah, I mean, I can speak to experiences too, where I was just like, man, I really like missed all the flags. And then I kept missing more flags and just kept going. And why are we surprised that we were in a bad place? Because I ignored everything. So I hear you, man. And kudos for recognizing it at some point and doing something about it.
Yeah, so what did we do about it? Unfortunately, well, not unfortunately. It was a good learning experience. But obviously, my happy place is deep in our legacy Rails application environment. finding all the pathways there, but I had to get really pushed myself, get out of my comfort zone. I had to pick up Kotlin.
Luckily, Kotlin is like a super friendly language, I think to get into, especially for folks who are coming from Ruby. Cause I think that there's like kind of like an emphasis on syntax on like syntactic sugar on like making the code actually like readable and look nice. Whereas that's not always the case with all JVM languages. So in that case, in that sense, it was kind of friendly.
Also at Ibotta, we were able to organize kind of a learning group. So there were lots of folks who were new to Ibotta or new to Kotlin who were kind of trying to solve these similar problems. So we made a study group. We found students. cool online coursework. We held each other accountable for making sure that we were making progress on those things.
And I started writing Conlon in the microservice to try to, like you said, push across the line. We need to get this across the line and then we can kind of circle back and deal with some of the underlying problems.
And that's important because as engineers, at the end of the day, we're trying to deliver value to the businesses that we work for and not just trying out new technology or optimize what technological solution we're implementing.
So just to clarify what you're saying, you kept Akka. You just pushed it across the finish line despite all your problems.
Yeah.
So that's how you resolve it. Okay.
So we have, we have problems with it and we decided let's get this thing. We're like, we're close enough, even with the problems, even with our lack of understanding, we're close enough that a total rewrite right now would put us back a long way and upset the business. So let's push it across the line. And then we can circle back and figure out what to do.
So delivering that value, even though it kind of felt yucky, even though we knew that there were things that were going to need to change over the long term, bought us some time and bought us some credibility in the organization.
And I want to circle back to the point you made about moving too quickly or too many changes happening at one time. And I think that's something that a lot of developers might start to experience soon with the whole removal of jQuery from Bootstrap 5. No! Not my jQuery! No!
Sorry.
So the idea is that Bootstrap 5 no longer has the jQuery dependency. So you might plan to upgrade your Rails application, the CSS framework from Bootstrap 4 to Bootstrap 5. And you might say like, hey, well, why we're making this change? Why don't we go ahead and rewrite a lot of our JavaScript that was also jQuery dependent? And so I would say instead of doing that,
do one thing, either first remove all your jQuery dependency within your application and just have jQuery be a dependency of Bootstrap. And then in another iteration, upgrade your Bootstrap version, removing then jQuery entirely, or do it vice versa, where you do your Bootstrap upgrade first, and then you do your jQuery removal from your application as a dependency.
But trying to do both side by side, it's too big of a task. for one person or one team to do right away. I would just handle one thing at a time and moving slower, you're going to say, okay, what broke this? Was it the new bootstrap framework that broke this or was it our jQuery rewrite?
So that way you're going to be able to identify a lot more problems quicker before they are reported to you by the customer. Absolutely.
I think the most experienced engineers, one of the best engineers I've worked with, his name is Justin Hart. He did the first commit on the monolith that I work on. And every time I work with him, that is always my experience. He's like, we're making this change. Here's what we expect. And we're going to do it. and then we're going to verify it, and then we're going to move on.
And I'm always like, oh, why don't we do this, this, and this? And he's like, no, no, no, just this one thing. And it's illuminating, right? It's illuminating. I think it's helped me as a developer in the projects that I'm doing now, realizing like, okay,
This thing that I want to do, it's actually like four different things, and I'm going to do the first thing first, verify it, and then move on from there.
If you or your loved ones are affected by the removal of jQuery from Bootstrap, why not check out the excellent course from devchat.tv, You Don't Know JavaScript Yet 30-Day Challenge, available with the link in the episode show notes. Were we scheduled for advertisement there? It just came out of nowhere. Just trying to keep the ship afloat, man. Come on.
So this goes along for me with... No, no, no, no, no, no, no, no, no, no.
No, we have to do jQuery now because this is really affecting me. I mean, this is terrible news. I literally can't write JavaScript without putting a dollar sign in front of everything.
You can still have a dollar sign.
You just need to assign something to the dollar sign, that's all. One of the reasons I started using Vue was because it has a little dollar sign in front of a data structure. And that kind of really makes me feel at home. Plus, you know, it feels like money. You know, when I type the dollar sign in, I feel like, yeah, this is going to make me a millionaire. I'm sorry, Luke.
So I guess while we're on JavaScript, I think another premature optimization, or rather I like to call them premature de-optimization, is creating a new Ruby on Rails application with React, the dash dash webpack equals React. Just thought I'd throw that in there for the ambulatory bash on React.
I do like stimulus. I do feel like we can let React go now.
Yeah, I mean, I don't know. Have you all checked out the... Basecamp's email, hey.com.
Oh, yeah. Yeah. What do you think?
Yeah, so I've been using it, and obviously I'm a Basecamp fanboy, but I think it's pretty amazing what they're able to do with just HTML over the wire for the most part and not relying on some of the frameworks that have come out recently. Also, I think it's super interesting, like the 2020 Ruby on Rails community survey. If you look, the one put out by Planet Argonne,
If you look at what JavaScript libraries people are using, so I was expecting that number one would be React. Maybe number two would be, I don't know, Vue or Ember maybe. But number one on there is jQuery. And it's like we're all in this jQuery world. And it's not bad. I love jQuery.
So for all of you who, along with Luke, are mourning jQuery, I'm telling you, stimulus, go check it out.
Try it and understand it.
I don't have an answer. That's fair. I thought I found it pretty easy. I thought it gave me everything that I felt like I was getting from jQuery before, which is a thing happened on the page. And because that thing... I mean, Jesse... Talk about event-based architecture here. Come on, this is exactly what JavaScript is, right? And jQuery especially, right? Stimulus is exactly that.
Event happens, do something else because this thing happened, right? Just saying, if you love that event-based architecture style, stimulus is that. It's just way prettier. And I found it a lot easier to use. And because I don't like dollar signs, I was happy about it.
Yeah, and Luke, if you want a bunch of different Stimulus.js tutorials, check out DrifterRuby, man. I have a whole bunch on there.
It's a big plug episode. Everyone quickly plug their thing. Do you know what? Actually, Dave, I have already checked that out and I still don't understand it. So I need to check it out again. It's definitely me. I tell you what the problem is, right? I've been leaning on jQuery for so long and I'm talking about 10 years, right? And it's not just that.
I've got my whole arsenal of weird front-end stuff that I can pull in. And replacing that big, long list of handy widgets I know will solve this problem is what I'm lacking. So, yeah, basic GOM stuff, fine. You know, ES6 and all the way. But if I want to do something weird... If I want to do something fancy, the whole point of stimulus is it doesn't have weird fancy stuff. It's clean.
So now that we've decided that Jake Ray is dying and hopefully our mourning period is almost over. It is dead. I know it's dead. I'm just finding it hard to cope. Okay, fair. So Dave, you earlier had me a little bit on this train and I kind of wanted to go back to it or whatever, talking about like changing multiple things.
Because there's so many places where we can find ourselves tricked or just like seduced into it, right? Where it just seems like the right thing to do. So the thing that I like to tell people, because it made total sense to me when I first heard it, was Kent Beck's, first you make the change easy, then you go and make the easy change, right?
And the important thing about that is there are steps here, right? Yeah, dude, engineering is all about taking those tiny steps and, like you said, testing, right? And we all know what happens when we change a bunch of things at once. And then we all do it. And, and then we, yeah, this is how we go down the road of mistakes. Engineering is this, it requires self-discipline.
And whenever we don't have self-discipline or whenever we, you know, just break it because we're human beings, bad stuff happens.
I almost feel like it's, I go into like meditative state almost. It's, it's, it's kind of like the flow state kind of. And yeah, There are times where it's like, I pick up a story and say, okay, yeah, this is easy. I can just kind of like half think about it and do it.
And when that doesn't work, which it almost never does, I have to like get into this like very focused, I'm making this one change verifying state. It's almost like the cartoon Avatar going into your Avatar state, if you guys know that amazing cartoon.
Yes, I'm familiar with it. Luke, I highly recommend it. What's it called?
Avatar and the Last Airbender. I think it came out like 10 years ago, 15.
Oh, it's the guy with the face tattoo.
He's got an arrow.
Yeah, exactly. You know exactly. I will check it out. I've seen that art too. He looks like he's really angry. Exactly. He's got no hair, right?
yeah bald head yep that's the guy yeah all right all right i'm a huge anime watcher too that's a that's a dangerous gap in my knowledge yes so anyway it's like you go into the state just oh jesse you were you were saying you go into the state when when the the first and it and unfortunately for me maybe it's because of my age or whatever but
it getting into that really focused state, actually that costs something, right? It takes like a little bit out. You can't maintain that state forever. There's like a mana pool almost of how long you, how long for me I can be in that extreme focused state.
And I think that when there are times where I look at a problem and I'm like, do it like, I almost like ask myself, do I need to be in this highly focused state to get this thing accomplished? And more often than not, the answer is yes, but my initial answer is no. And so I end up wasting time by not doing things as systematically as they need to get done. Am I alone on this?
Nobody else goes into Avatar State when they code?
I find that people talk about this flow state, and I was not a believer for a while until I got something really hard to work on. And it's those problems where you're at the kind of limit of what you can do when you're thinking, can I actually write this? That is when I find you get these kind of periods of intense concentration.
And obviously, you don't want to be, you know, at the edge of your ability all the time. You want to kind of line up the nice, easy jobs or things tend to go disastrously wrong. But one thing I have found is that, you know, when you're doing these really hard problems and you're like, can we do it? You know, is it possible? You always have to come back and redo it.
Once you prove that it's possible, then you have to hit it again, and then you come up to the one. So all of the stuff I've done in a kind of state of intense concentration tends to get thrown away. But it moves you forward down the road.
I always call those my naive implementations. And it's totally cool to write really terrible code during naive implementation. That's not the point of it. The point is to get from having nothing to having a working thing. The important thing is that after your naive implementation, you decide whether it's worth refactoring, when you're going to refactor, all that kind of stuff.
Yeah, I love that idea of a naive implementation. And I think the Akka microservice was our naive implementation. We didn't consider the scale at all. We didn't consider the makeup of the team changing.
And as a result, when we went back to fix the naive implementation, when we wanted to get rid of the double loops or whatever, we ended up moving to Java, so from Kotlin to Java and from Akka to Camel. And the reason that those tools made sense is because they were very common in our company. And we didn't have to reinvent the wheel.
We weren't seeing errors for the first time in the entire company. It was sort of a knowledge base to draw from. So the current state is that we have Java Camel Spring microservice talking to the Rails monolith. It's very stable. It's performing super well now.
And, yeah, I mean, getting... I guess a question I have is like, how can we speed up the process of getting from our naive solution to the more learned solution?
I think, in my opinion, if I had an answer for that, I would be... I would be a super wealthy person because in my experience, usually what happens is when you get to the point that you're calling something a naive solution, right? It's usually because you recognize that there's a problem with it, right? And one of the things that's going on, I feel like
is that you're kind of burning yourself out on the problem at that point. When you sort of have this realization, you're also at the point that you're kind of burning yourself out on this problem. So if you as an organization don't have the resources to sort of like swap out people, you know, bring somebody in that's fresh, things like that, right?
Or kind of go back to a huddle and, you know, somehow become re-energized. Like all the things that we can do are almost all like social interactions, not engineering interactions, right? And so if you have a culture where you can kind of deal with that,
I feel like people can kind of turn around and refactor and make reasonable choices or whatever but if you don't like I feel like it's really hard it's really hard for you as somebody who just burned yourself out pushing something across the finish line that was really hard to then turn back around and be like I'm throwing you out and I'm gonna redo this really hard problem again right like that's just a really hard thing to do I think it's interesting that you say that because at the point that we made that decision the problem didn't seem hard anymore
At the point that we were like, you know, we can just do this in Java and Camel, we had wrapped our heads around the problem enough that we really felt like it wasn't hard anymore. And maybe that's what it takes.
You also said a really important word there, which is we, right? So one of the things that you did as a company to recover from this problem is you went back to this huddle and you said, hey, I made some mistakes, right? And and then your team said, that's OK, Jesse, we as a team are taking ownership for this. Right. Like and we as a team are going to fix this. Right.
Like that's what a lot of that communicates right here. And when you do that, like that's one of those reenergizing moments or, you know, things. And then those decisions become a lot easier in my experience.
It breaks my heart every time. Every time I have a git commit with kind of more lines removed than added. Oh, man.
Those are the proudest days.
Right? I know some people like to take the code out and they're like, oh, I go knocked out, you know, loads of that repeated code. I just think, why couldn't I get it right the first time?
That's interesting. The person who taught me how to code... one of the first pieces of advice that he gave me was that you should write a lot of code and throw a lot of code out. And that's how you'll get better at coding.
That is very, very sensible advice. Very sensible advice. And by that standard, I've got amazing at coding over the years. The context is a big mistake. Now, just to be clear, this big mistake has had a happy ending. So this isn't the kind of big mistake I got fired. This is big mistake. We pulled through it and it all worked out. Am I right?
Thankfully. Yeah, so we did a couple of things I think that helped us. So the first is, as John said, we were open about these things. We didn't try to hide that things weren't going as well as we had wanted them to. And I think that Ibotta has a pretty strong culture in the sense that We're not trying to throw people under the bus in engineering.
If something crashes, it's not who's going to get fired. It's like, okay, how do we learn from this? This is a mistake that cost us some money. How do we make sure that that money is actually teaching us something? So that was part of it. And then I think also we did a good job of communicating to external stakeholders.
We communicated to the finance team who were kind of one of the main consumers of the data that we were producing. and really went through in detail. Here's where we're at. Here's the timeline, like updating them. We were checking in with them all the time and just keeping expectations in line, I think really helped us out.
So even though we delivered a little bit later than I think we thought we would at the onset of the project, because we were able to communicate that, we were not fired. And yeah, I mean, not only that, We've hired more people. We're still hiring.
And if you're thinking about getting into the mobile coupon space, there's a ton of really cool problems, even if you're not passionate about mobile coupons. And you might get to talk to me in the interview process, which will be fun.
Speaking of interviews, I see a smooth transition there. Speaking of interviews, I understand that you have a project which you call Mina Swan interviewing. I have no idea what Mina Swan sounds like. I see a lot of people saying it in the Ruby community, but it's one of these weird in-jokes I think they have in the Ruby community.
it's nice and so i know i know i know dry humor luke never pick up on your sarcasm good lord i'm just over here like like dying everything and that's the uh what was it dave sorry it was seriously yeah go on uh matt's it's nice and so are we
Yeah, what a best thing about Ruby's community. It's a really great language and it's a really strong community, really great events, even though obviously the events this year have been a bit difficult. So how are you carrying that culture, that tradition over into the interviewing process? What's your gambit?
Yeah, so maybe everybody has experience or a lot of folks have an experience in, let's just say, a not-mean-us-one interview, an interview that maybe is a little more hostile than we'd like.
And I think a lot of us have experienced kind of broken interviews where it feels more like the person on the other side of the table is trying to prove how much smarter they are or how much better they are at coding than I am. And that's not nice.
Another thing that's not nice is asking someone to do an inordinate amount of work outside of work that's not paid in the form of like a take-home project. So I've done take-home projects that have taken me an entire weekend, multiple days, and that's uncompensated work. And that can bias your process against people who have outside of work commitments like families. And I'm just,
you know, who don't want to be working all the time. So I thought it would make sense to kind of take the best part of the Ruby community, this idea that Matt's is nice. And so we are nice and apply it to interviewing. Let's like actually be nice to the people that we potentially could be working with. And, you know, the pandemic has been terrible in so many ways.
But it did offer us this opportunity to kind of dramatically rethink what our interview was going to look like. Because we're not coming into the office. Everything has to be remote. And basically, our HR team and our leadership were like, how can we do this? We've only been accustomed to bringing people in, asking them super tricky things that they have to whiteboard.
How can we translate this to a remote interview? And this is what I proposed. And this is like... what we landed on, which is an interview not meant to trick the interviewee. It's an interview meant to simulate what the first couple days of work is going to look like. And it's supposed to give us as an organization a sense of how much we would enjoy
this person as a colleague, how successful they'll be. And the message that we're always trying to send is not, hey, I'm so much smarter than you because I understand recursion or because I understand how to do whatever this type of model, this type of data structure.
The message is you're going to be successful on our team and you're going to like working with us and we're going to like working with you. So the way that we do that is basically by giving the person a sample of code from our domain And it's highly simplified. And we ask them to just read the code. And we say, what is this doing? What do you like about this code?
What do you not like about this code? And it's not a bug-finding adventure. We're not asking them to find where an error is going to be secretly raised or why a test is failing. You can't really do that in a 20- or 30-minute conversation. We want to hear how this person would approach an alien code base, which is what their first task is going to be on the job.
Then we present them with some data, some award events from our system, and we ask them to manipulate that data with a pretty simple algorithm. We even tell them what the algorithm is, and we ask them to code it. And we say specifically, We don't care about the answer here. The answer is not interesting. We just told you what the answer is. We actually want to see what it looks like when you code.
Now, what's your approach? Are you systematic? Are you making guesses about what should happen and checking yourself? Those are the things that we're looking for and we're not looking for some to see. Do you know this random algorithm from your from your computer science education that you'll never use as a web developer at Ibana. So those are the big pieces.
And I'm stoked because I got invited to talk at RubyConf, which is coming up in November, where I'm gonna be kind of outlining this. You all got a preview of the content there, exclusive to Ruby Roads.
Awesome. I'm definitely curious, as you guys implement this, when you kind of come back and say, well, this works and this didn't work, or this is how we sort of had to come up with a way to Because this is a subjective thing, this is how we came up with a way to make this more objective than it originally was. And that helped us. Definitely interested in seeing this.
All of the pain points that you're talking about, all this homework that we have, all these coding challenges that we do. I feel like as a community, we talk about how aware we are that they don't work. But we don't have alternatives yet. And so people continue to use them regardless because they're like, well, I don't know what to do.
And people who are lost, you know, just run back to where they came from a lot of times, right? It's like the squirrel. It's like in the middle of the street and a car is coming. It's like, shoot, I got to go all the way back. So I kind of feel like we do a lot of that still. Yeah, there have been various things like talked about.
I remember, I think I've heard various people from ThoughtBot on various podcasts, like talk about the fact that like, They have you come and spend like, I don't remember if it was like a day or whatever, a half day, but like a lot of time with them pairing, right? I know that Bootcamp does that as well. Or Basecamp, sorry. Wow, whatever. Basecamp does that as well or something similar.
Though, actually, I remember they had an article about they did some homework or something recently. Whatever.
They pay for the homework, though.
Yeah, that's okay. Yeah, I mean, there's people that are like exploring around these edges, but we don't really have like a good way forward, I feel like yet still. So I'm super interested to see how this turns out.
Yeah, I mean, so far, obviously hiring has been slower than typical for Ibotta, but we are still hiring. And so far, the feedback from candidates has been really interesting because we'll get unsolicited emails about how much people like the process and how they hope that this is the direction that the industry goes in.
a lot of folks who are interviewing with us are interviewing at other places as well. And I see this as like a competitive advantage for us, right? Does this buy us like $10,000 in salary or $5,000 in salary or something like that?
Like, does this make our company more, you know, once you get to a certain point, there's like diminishing returns on all these different levers that a company can offer to a developer. And I think that this is maybe an unexplored lever or a lever where there's a ton of room to gain. And when, Someone goes through our process and comes out feeling like, hey, I'm going to kick butt there.
The people there are super nice. They didn't make me feel like an idiot. And they contrast that with other interview experiences that they've recently had. Hopefully that will play in our favor.
That is certainly not the current approach to coding interviews. I've been hearing recently more people talking about how to make coding interviews harder. If you go on something like Hacker News, you know, they discuss which
framework they need to insert under the fingernails of potential applicants and i hear if you go to facebook now and apply for a job then they uh you have to bring along your phd in computer science they uh then burn it in front of you mix it into old coffee and make you drink it as part of the interview process so it's really i think it's really something that the tech community needs we have been doing a few episodes about trying to make community more inclusive trying to bring in people trying to keep people in
And that sounds like a really great idea for a talk. You know, why don't we be nicer to people, try and get the most out of them during interviews instead of subjecting them to torture?
I love that image of shoving a framework under someone's fingernails, Luke. But the thing that I noticed, and I've done a lot of interviews that I've done, I would say easily 200 interviews, over the past two years. We do a ton of hiring, so it's crazy.
And the thing that I noticed is that when we were asking people to whiteboard, to answer these tougher algorithm questions, I didn't always see a one-to-one correlation between the people who crush that kind of question and the people who I really enjoy working with in either direction. There are false positives and false negatives.
And I just found there to be way too much noise in that type of diagnostic tool for me to really make a strong decision that I felt confident with that proved out over time. And I'm hoping that this is a remedy for that.
Yeah, I mean, you always have to check to make sure that whatever test you're giving, right? Like, you have to decide what you're testing for. And you have to say, is the question that I'm asking actually testing for the thing that I'm testing for? And too often, we jump to this conclusion that,
that oh yeah this sort of this sort of is related therefore it'll let me know when really you're just you're making a subjective judgment call again and calling it objective so yep it is it is a thing that's very painful and maybe maybe we should we should focus a little bit on this in the future and have a more in-depth chat about this. I feel like that would be an interesting subject.
Yeah, there was a RailsConf talk last year. I believe the guy's name is Eric. I'll throw it in the chat in a second. He's from TestDouble. And he talked about, TestDouble is like a Ruby consultancy based out of Columbus, I believe. And he talked about their hiring process
And more in the sense of our goal is to build a process that lets candidates show off their strengths and as opposed to helping us find their weaknesses. And that talk really informed our process at ibotta. We're trying to find people's strengths and trying to figure out where they're going to be able to make the most impact in our company.
Yeah, I have gone for the past, like, I don't know, four or five Rails comps. I've gone to basically every single talk that was sort of like related to the subject. I do, I care about it quite a bit. And I just kind of hope that we get to a better place. It's...
Yeah, like I said earlier, my feelings on this, I think, are basically what I said earlier, that I think that a lot of people are thinking about this. And I just don't think there's a clear answer yet. But I'm super interested in everything that people are trying. Experimentation. That's how I think we're going to get a little better.
Yeah. Well, hey, it looks like we're coming up on the hour, so we need to move things along. Jesse, if people want to get in touch with you, where should they go online and look? I would say you can find me on Twitter.
I tweet from PlanetEfficacy. So if you can spell that, you can find me. And you'll find a lot of interesting Ruby content, political outrage, and Vermont progressive rock. on that Twitter stream.
Awesome. Well, let's go ahead and move over into some picks. Luke, do you want to kick us off?
I would. My first pick is a version of Windows 10. A bit of a strange thing to pick on a reverse Pogfiles, but this is called Windows 10 Ameliorated Edition. I found it featured on a popular YouTube channel not long ago.
And this is a version of Windows 10 called a spyware taken out, which also incredibly boosts the responsivity and performance because it's not doing any async network calls back home in the background, if you know what I mean. This is the first operating system I've ever found where to download it, I had to get a BitTorrent link from a Telegram channel.
So if you want some excitement in your operating systems, check out Windows 10 Ameliorated Edition.
Never seen anything like it. I just wanted to confirm something here really quick, Luke. Is this a legal version of Windows 10?
It's got quite a big section on legality on the website. Okay. The website does make it clear that you do need to have a Windows 10 license in order to legally use the Windows 10 Ameliorated Edition. But, I mean, come on. BitTorrent link through a Telegram channel? Wow. That's quite a... Anyway, it's quite chubby. It's a cool little project. There's lots in the FAQ about their rationale behind it.
And it does fly when it runs. It's really quick.
I heard there were some issues with it with not being able to do certain things because it can't call home to Microsoft. Even some like simple rudimentary things that you would think that would be possible, but just kind of breaks it.
Yeah, I mean, you could call it Windows 10 lobotomized. I mean, there's barely anything there at all. But it's a lot of fun. It flies doing nothing.
Awesome. Well, John, you want to do some picks?
Yeah, so I have a different pick this week, too. By the time this comes out, I have no idea if people are even going to still be playing this game. But I have been playing and streaming and absolutely having a blast playing, I guess that's redundant, Among Us.
So if you're not familiar with it, if you're familiar with like Werewolf or Mafia or kind of games in that nature where you, there's another game that people have like compared this to or whatever, where you like sort of have like
like townsfolk and then you have like people that are pretending to be townsfolk that are like trying to kill everybody off and your goal is the townsfolk are to try and find the uh well in this game the imposters among you so this is like set in space or whatever but yeah it's an absolute blast i i really am not sure what to say about it other than it's like kind of like a great social game
yeah it's it's pretty awesome so i highly recommend checking it out i definitely have been doing a lot of streaming of it lately so yeah i'll jump in with a few picks first pick bamboo flooring so i love bamboo and instead of the crummy old chair mat the little plastic thin plastic that I was using for a long time. I finally went to Home Depot and got some bamboo flooring.
And I glued that to a piece of plywood. And I now use that as my floor mat on my carpeted floor. So my chair can just slide around on it. It's really, really cool. And it's been a life-changing event here. So it's pretty awesome. And the second thing that I'd pick is the Elgato Key Light. So I got those for my streaming setup and they make a huge difference.
So you won't be able to tell on the podcast, but this is with my lights on and this is with the lights off. Makes a really big difference in quality. So having proper lighting on doing any kind of video work is an absolute must.
Can you do that again, Dave? Just so everyone can see on the podcast?
Yeah.
I got to say that is quite striking. Is that the same company that did the shortcut bar?
Yes, the Stream Deck.
Yeah, that's a pretty cool thing as well.
Big El Gato fan. And Jesse, do you want to jump in with some picks?
Yeah, I've got a couple of picks, different categories. So first is a talk that I watched recently that maybe you all have seen already, but I recommend revisiting it. So Sammy Metz's RailsConf 2019 keynote, which is called Lucky You. And I think it's especially timely as we approach election season. And thinking about equality and issues of inequality in our country.
So I highly recommend that talk. It's amazing as all Sandy Metz talks tend to be. Then I have, when the pandemic started, we started working remote. I made a small investment in my work from home setup that I highly, highly, highly recommend. I went out and I got an ErgoDox split keyboard, having some wrist pain on the keyboard I was using.
So I know keyboards can be a whole other podcast, but check out ErgoDox. They make a really, really, really good product that's worth the price. And I've been a huge fan of it since getting it. And then my final pick, there's a book that just came out. It's a guilty pleasure. It's called The Trouble with Peace by Joe Abercrombie, a British gentleman.
And it is in the grimdark genre of fantasy literature. And I highly recommend it.
Awesome. Well, thank you for those picks. And just be sure to post them into our chat section here so we can include them on the show notes. Well, Jesse, thank you for coming on today. It was a lot of fun. I love these kind of talks where we can just humble ourselves and talk about past mistakes and what we learned from them. So thanks again.
This was awesome. Really interesting.
I listened, I, a long time listener, first time guest, and this was, this was awesome. I appreciate it.
Yeah. Thank you for coming.
Right.
Well, that's all for this episode, everyone. Thanks for listening.
Take care, everybody.