The Changelog: Software Development, Open Source
The best, worst codebase (Interview)
Wed, 18 Sep 2024
Jimmy Miller talks to us about his experience with a legacy codebase at his first job as a programmer. The codebase was massive, with hundreds of thousands of lines of C# and Visual Basic, and a database with over 1,000 columns. Let's just say Jimmy got into some stuff. There's even a Gilfoyle involved. This episode is all about his adventures while working there.
What's up friends. Welcome back. This is the change log on this show. We talked to the hackers, the leaders, and those working on the best worst code bases. Oh my gosh. Today we're joined by Jimmy Miller to discuss his experience working with a legacy code base. at his first job as a programmer.
The code base was massive, hundreds of thousands of lines of C sharp and visual basic and a database with over 1000 columns. Let's just say Jimmy got into some stuff. There's even a Guilfoyle involved. And today's episode is all about Jimmy's adventures while working there. A massive thank you to our friends and our partners over at fly.io. Yes, that's the home of changelog.com.
Launch your apps, launch your databases, and even launch your AI near your users. Fly is the public cloud built for developers who ship. Launch your app in five minutes at fly.io. Okay, let's do this. What's up, friends? I'm here with a new friend of ours over at Assembly AI, founder and CEO Dylan Fox. Dylan, tell me about Universal One. This is the newest, most powerful speech AI model to date.
You released this recently. Tell me more.
So Universal One is our flagship industry leading model for speech to text and various other speech understanding tasks. So it's about a year long effort that really is the culmination of like the years that we've spent building infrastructure and tooling at assembly to even train large scale speech AI models.
It was trained on about 12 and a half million hours of voice data, multilingual, super wide range of domains and sources of audio data. So it's super robust model.
We're seeing developers use it for extremely high accuracy, low cost, super fast speech to text and speech understanding tasks within their products, within automations, within workflows that they're building at their companies or within their products.
Very cool. So Dylan, one thing I love is this playground you have. You can go there, assemblyai.com slash playground, and you can just play around with all the things that is assembly. Is this the recommended path? Is this the try before you buy experience? Look, people do?
Yeah. So our Playground is a GUI experience over the API that's free. You can just go to it on our website, assemblyai.com slash Playground. You drop in an audio file, you can talk to the Playground. And it's a way to, in a no-code environment, interact with our models, interact with our API to see what our models and what our API can do without having to write any code.
Then once you see what the models can do and you're ready to start building with the API, you can quickly transition to the API docs. Start writing code, start integrating our SDKs into your code to start leveraging our models and all our tech via our SDKs instead.
Okay. Constantly updated speech AI models at your fingertips. Well, at your API fingertips, that is. A good next step is to go to their playground. You can test out their models for free right there in the browser. Or you can get started with a $50 credit at assemblyai.com slash practical AI. Again, that's assemblyai.com slash practical AI.
So we're joined today by Jimmy Miller, host of the Future of Coding podcast. Jimmy, you wrote the best, worst blog post.
It was amazing. Nice little play on words there. Yeah, I recently wrote a blog post talking about my first job in programming, and it's the best, worst code base I ever worked in. And so, yeah, it was a really fun post and kind of blew up. I mean, you know, way more than any blog post I've written. Got like primogen on YouTube, like doing a video on it. It was really cool. Really neat to see.
Top of the hacker news, all that stuff.
Yeah.
It's really fun.
Definitely resonated with me. In fact, I was like stopping to check the technologies that you listed because I literally thought maybe I had been on the same code base. And then I was like, I was all C sharp and VB. I was like, okay, mine was over here in Ruby land. But I was just like bringing up PTSD or something, you know, because... I've been in a code base like this.
Well, first of all, let's lay some groundwork here before I start to get into the details, because this was your first job coming out of college, is that right?
I did not go to college.
Okay, coming out of schooling.
Yeah, yeah. It was a couple years after high school, but yeah.
And successful business that you went to work for, I assume?
Yeah, it's a big credit card processing company that's now been merged or bought out or something. But, you know, pretty, pretty big company. This was like a small branch of it that was like the customer support center. But, you know, fairly in the grand scheme of things, a medium sized company, but fairly large, you know, for actual software development.
Right. Yeah. And so lots of money being made, of course, credit card processing, a core piece of the world's infrastructure. If you're getting fractions of a penny of every transaction, I mean, there's just a lot of money coming in and money hides problems, right? Like the success just hides problems. And this thing had so many problems. It's just hard to fathom that it operated.
Yeah, it was honestly pretty crazy. Going from the only code I had ever seen was code I wrote myself to diving into this kind of code base where we got hundreds of thousands of lines of C Sharp and VB with just crazy database configurations, all this wild stuff going on. And just realizing like, oh, this is what real world code looks like. And of course, I had no other point of comparison.
I now realize... It was a little unique, a little bit of craziness there. But yeah, it was one of those things where you think if you're a successful company, the code's got to be good. And I found out that that's not really the case.
There's something about the naivety also of being fresh out of school or young to the industry. I think I told this story before, which I had a similar experience where I inherited some bad code, but I didn't have that perspective and just knowledge enough to realize it was bad code. I thought it was good code and I was a bad programmer. And I probably, I mean, I was.
But still, I gave it too much credit because I think this must be what good code looks like. It's so hard to understand, you know?
Yeah.
And it took me a couple of years of just maintaining that. And thankfully, I had autonomy. So I just did it by myself, like slowly changing a thing here or there without major interruption from bosses or anything. Then I realized what good code actually looks like and that what I had inherited was actually just clever but terrible. And oftentimes clever is terrible.
And there's some clever stuff in this code base you're talking about.
Yeah.
Things that you're like, oh, that's kind of clever, but it's also so dumb.
I think that was one of the lessons I had to learn was just how clever you can be and how much you can solve a problem with the most complicated code you could possibly imagine. And yet, for the end person, they don't care. It doesn't really matter. But for me, I don't know. I've never been a business-y person. I like coding because I think coding itself is really interesting.
And so for this first job, it was kind of a shock of like, this is what I have to deal with? this ugly grossness. But I think that that's changed a bit. I look back on those moments quite fondly. So yeah, maybe I should talk a little bit more in detail about... I'm assuming most people haven't read the blog post or whatever. Happy to fill in some background of what was this code base look like?
What are the weird things going on in it? Or...
Yeah, absolutely. I think we can drill in on specific aspects and just enjoy them as we do, but you can definitely paint with broad strokes in terms of, I already said it's a big visual basic C-sharp thing. The database was massive. It had lots of columns.
You go ahead and fill out some of the big picture aspects of what this thing was, and then we can dive into some of the details because there's a lot of just enjoyable tomfoolery going on.
Yeah. So let me paint a little picture of like the, you know, the company. So this is a big, you know, like I said, a big credit card processing company, but the actual office I'm at is almost all staffed by customer support people. There's just this like one room that's developers, probably like 80 developers or so in this one big, huge open office. Wow.
And we're all writing this software for these customer support people. It's totally separate off from the rest of the credit card processing business. This is all bespoke software built up over the last 10 years. This is like... 2012 is when I joined this company. So it was a little bit behind the times, even for then. Very stuck in the past of how software is being done.
But I don't know, it's your typical, the customer support people are all really serious, have to dress a little nicer, and the developers are all kind of chilling, shooting each other with Nerf guns and being a little bit more wild. Yeah. But the code was the kind of part that everyone wanted to ignore. The stuff I was on was... I started as an intern.
It was like the legacy code base that the professional developers didn't want to touch anymore. There were all these teams who were going and redoing the big rewrites. And the interns and a couple developers were kind of shoved on this old legacy project. That was just... So the database itself, I talk about in the articles about we ran out of columns.
The database is this massive database with the merchant's table, which has 1,024 columns because that's the most you could have in SQL Server. The code base is hundreds of thousands of lines of C Sharp and VB. And the reason it's kind of split is they decided halfway through to change from VB to C Sharp.
But the whole way it worked was session state got stored in the database every single time you changed pages so it could swap back and forth between the VB and C Sharp world. It's this crazy bespoke IIS setup that takes days to get running on your machine.
Mm-hmm.
The whole thing was just, yeah, kind of duct taped, terrible code base, every JavaScript framework you could possibly imagine. And the task as an intern was, here's a big list of bugs and features that we don't want to actually spend developer time on. This is your job now.
You got to ask yourself, how does a code base get to this point? Like, is it because there was no leader? Is it because no one cared? Is it because it was sort of siloed off the seemingly primary application, which ran the processing? This was, as you said, customer support.
Maybe it's a sidecar to the business and less important, but you talk about sales folks putting their wins in there and logging in because you've got to... The calendar and stuff like all these, like, how does that even happen? Is it because there's no leadership? What do you think?
Yeah, that's a good question. So like in some ways it's, you know, less important because it's customer support, but it's also, yes, the sales organization is kind of in this code base, right? Like that, that really matters, but also shipping out payment terminals all happened from this site, or at least a lot of them happened from the site.
They might've had other locations, but like, so shipping was also a big part of this. So it's not like it was like completely to the side. Based on my time there, obviously, and like all the stories I heard, it's really leadership conflict. Constant changes in leadership, constant people coming in and out at the top level, different directions changing. One of the things that was a really...
The kind of annoying aspect of these code bases was the names for every product had changed a hundred times. The names for every, like, there's like this big sales hierarchy and it really, really mattered what the sales hierarchy was. Like there's the regional manager and the district manager and the, you know, all these different terms. And they were just abbreviations in the code base.
It'd be like the DM, the RM. But those abbreviations had changed over time to mean the same things. It would be like district manager versus direct manager. And so you'd actually have to look to know what those entities refer to. You'd have to look at source control and then go talk to Munch, who was, as I put, the resident shaman, and ask him, what things in this year did this refer to?
Wow.
So it was just this constant change, right? And constant change in direction, constant change in leadership that just made the code base go in so many different directions as well. I think, you know, Conway's Law ends up winning out on all code base organizations.
When you think about it, you knew Munch, obviously, but if you didn't know Munch, would you think he's like a Dwight? This reminds me of The Office in a way. Instead of paper, it's a software package, you know, or a program. It's like just getting the job done, basically, is the mission.
Yeah, no, I can totally see that. I don't think I would call Munch a Dwight. Munch was a very charismatic person. He was somebody who has management wanted him to be in charge of so many times and he just refused to go be promoted beyond that. So he was like the de facto leader of so many things, but his job title never reflected it.
I think his job title was just like systems analyst or something like that. He wasn't even technically a programmer by the org chart, but he did so much. So I could definitely see the Dwight comparison, but I think that would be selling him a little short. He was just a really kind-hearted, really nice person
At any time you needed help on anything, he was the one who was willing to like sit with you and explain it. He was a great storyteller. I mean, I think Shaman is the best descriptor I can give for Munch.
Was his name actually Munch? I mean, who names their kid Munch?
Okay, so his name was not actually Munch. Apparently there are only two people in the world that know why his nickname is Munch. These were two high school buddies, and he tried to get rid of the nickname when he went to college.
He tried to just go by his normal given name, but then there was somebody else in his dorm with the exact same name, and one of his friends knew him as Munch, and they just said, oh, we can just call him Munch, it's fine. And then a second move, he moved to like a new town after college and tried again.
And yet like some, he ended up running into an old buddy who called him Munch and then it like spread. And so even his wife apparently didn't know how he got the nickname Munch.
That is hilarious. Some names just find you, you know, you just can't, you can't escape them. They just stick.
Yeah, I remember the day where we got a new system for being able to, like, naming. Like, we got a new, like, AD setup or whatever so that, you know, kept your emails and your names or whatever. And it turns out that it had a little lower permissions requirements than the last one. And Munch went and changed his name and everything to Munch.
So instead of actually having to be like, oh, yeah, that guy, that's Munch, now in the system was properly Munch, which was fun.
He finally embraced it, huh?
Yep.
Nice. So let's go back to these columns.
Okay. Yes.
Okay. So there was a merchant's table. It hit the max number of columns, 1,024 at the time. Apparently, was it SQL Server has got more memory now? You can now add 4,096 columns. Yes. Yeah, yeah. Good for them. But because it's arbitrary, I mean, it's not arbitrary, the memory constraint is
force them to either stop making new columns or create a second table in which you can just shove some more columns. That was the choice that was decided. So now there's merchants and merchants too. And I assume every time you look up a merchant, you just got to pull both tables in, don't you?
Yeah, I mean, it depends on what columns you're grabbing, right? You have to know what things are where. And most applications, I mean, most of these columns were completely pointless. I am sure if you actually looked at them, they were completely not used for eight years or something, right? It's just like somebody decided, I'm just going to add another column.
All of our SQL schema was not actually in any source control. You just had to submit forms to the DBAs who would manually run them for actual like stored procedures of which there were hundreds or thousands. You had to like manually check them out in different environments and then write your name like I am editing this. Please don't edit it.
Uh, and then you would like leave a little like change log of what you did and you'd end up finding out like someone didn't do it in the lower environment. So by the time you get to production, it's wrong and all sorts of, uh, fun things there. So, you know, adding a column, as long as you can get one DBA to agree, you're good to go.
Which they did thousands of times, apparently. I just wonder, how can you possibly have that much information about a single entity in the world? Obviously, there's probably a lot of foreign key relationships, which is probably a lot of those columns. But I mean, how much can you know about a single merchant? Like more than 1,024 things matter.
Yeah, it was. I think, you know, a lot of it were was things like, are they involved in this promo? Do they have this kind of equipment? You know, there was all sorts of different iterations of this product or like, you know, who is their salesperson? And it's so there there was honestly like the domain itself was very complicated.
I never got a full picture of all of this domain of what's going on. Now, as we now know, of course, and even then it was happening, like Stripe has simplified payments to like a nice minimal thing. And I actually remember at the time, there was a group that was supposed to be doing open source work and trying to take on Stripe. And I was so excited. I was gung-ho, 20-year-old, first job.
I was like, I'm going to join the cool group doing open source. And they had some Python code that looked straight up like C Sharp code. And I made a PR to fix their indention, and they got really mad at me. They're using tabs instead of spaces. You know, it's Python, like four spaces. And yeah, no, it did not go well.
But yeah, like I think, you know, for me, a lot of this was just like I didn't know anything about like the the broader tech ecosystem. Like I, I never even realized that I could do this as a job. Like growing up, I just kind of learned programming on a whim and had no idea that people, the stuff I was doing was what people actually did.
Yeah.
Yeah. Uh, The way I got into programming was a bit weird. I grew up relatively poor, not always having food on the table poor, but I still have a roof over my head, still electricity, the US poor. Obviously, there's people who have it much worse in the world, but We did have a computer, luckily.
We were gifted a computer from some family friends, but my brother would always hog it, my older brother. And so one day he decided that I got my own computer because he found one by the trash with mud on it. It was this old Dell with Windows ME on it. And he just gave it to me and said, this is your computer now. This is the one you have to use. It didn't work.
I didn't know the password for Windows ME. But luckily, I had a friend whose dad was a system admin. And he had mentioned Linux to me one time. And so I got Ubuntu running on it. And I was 12, had no idea what I was doing, just burning CDs, trying to get some software to work on it. I ended up getting Ubuntu on it. I ended up getting wireless card drivers working for it.
Wow, that's an accomplishment.
Yeah. In Diswrapper was this like taking Windows... It didn't have a wireless card. I thought, okay, I can buy this and install it and it'll just work. But In Diswrapper was this way of like wrapping Windows wireless drivers and trying to make them work for Linux. And I now know what I did was I compiled from source In Diswrapper by burning CDs to get the dependency tree.
It was the only way I had to transfer data from the computer to... between computers and so i like burned these cds and compiled from source indiswrapper and i got it working and that was when i was just hooked like this was so exciting that i got my own computer working by doing a bunch of magical incantations i didn't understand at all
That's amazing. So you would download them on your brother's machine?
Yeah.
And you would rip them to a disc, to a CD, which is a one-time. But yeah, you pretty much have one shot at it, right?
Oh, yeah. I didn't have readable, writable. I luckily had gotten some, yeah, right once.
And you sneaker net that sucker over to your machine.
Yep.
And then are these just tire balls or how did you, do you remember?
Yeah. Yeah. I, when I, when my brother was gone to a friend's house, this is when I would go and, you know, sneak on and go do this and get things working. Uh, so yeah, I mean, he was, he was nice. If I really asked him to let me have computer time, it's like, he completely kicked me off, but you know, your older brother, right?
Like you, when you're, when you're that age, it's like, ah, I can't, I can't bother him too much. Uh, so yeah, I got it working and that was my computer for. probably five years. It had like 128 megabytes of RAM. It was, you know, at that time, it was quite a bad machine. But with Linux, it ran super smooth. It was great.
Linux is the best. So this was your first gig, really, this place. And you said that you were self-taught as a programmer. So how much experience did you have before this job doing programming to know that this was bad? Like, not the way.
Yeah. I had only really, I mean, I'd written a lot of code in my spare time. You know, I did a bunch of like projectoiler.net stuff. I don't know if you all are familiar with it, but it's like for anyone who's not listeners, it's a bunch of math problems that you do have to use programming to solve. So it's a bunch of like number theory kind of stuff I'd solved.
They're like they get really difficult, really fast. Like I think I've solved all of them I will ever be able to solve in my lifetime because like as they continue on, it's just like graduate level math. And so I had done a bunch of those. I had played around with like Mozilla had had a new extension method called Jetpack that they were playing with.
I had made some of those and released some and people used them out in the wild. They were all like super small programs. I had written a program for my school. I had a great programming teacher. Miss Johns was her name. She was fantastic. I love her. She did not know programming, but she was a great person who would teach from a book and recognize that I knew more programming than her.
So I just was a tutor in the programming classes. I took the Java AP exam, never took the class, got a five on it. I did a bunch of stuff in my spare time. So I learned Java, I learned Python. I just loved programming. It was my main hobby. I'd skip my calculus class to go program. That was the kind of... Failed a lot of my classes because I would just skip them to go programming.
Well, it paid off.
Yeah, Miss Johns was actually the reason I got this job. A few years after high school, she had reached out to me and said, hey, there's this company looking for interns. And she told them about a story that I'm happy to tell about a time that the Secret Service busted in my door for hacking.
I do want to hear that one.
Uh, and they were impressed enough to give me an interview despite, you know, not going to school. So, right. Um, so yeah, happy to tell that story, but, uh, I also, you know, no, we're, we're here to talk about the code base.
So no, uh, secret service knocking down the door, I think is a worthwhile diversion. Don't you think Adam? Yeah. Is you got more to the story? What's the story? I mean, were you, I mean, you, you shall it, it deep.
Uh, okay. So yeah. So for people who don't know, like the secret service also deals with like, uh, internet security. You think of the secret service as just being the president, which is what I thought when they busted in my door. I worked at a local or regional grocery store as a cart pusher when I was in high school.
And the system to check your schedule was the most convoluted, annoying system you could possibly imagine. You had to get on a VPN. Then you had to log in four times. And then you could finally traverse these links to go check it. And they would publish the schedule Saturday night for the Sunday morning.
And I just got so annoyed with having to spend like 15 minutes logging into this system that I was like, I'm going to write a program. It's going to do all this for me. And it's just going to email me my schedule every week. So I go to write the program. And I start, you know, it's a simple little Python program making some HTTP requests.
And I was like, all right, let me figure out if I don't send a password, like what error I get. And then I can, you know, continue on from there. I got the schedule. I didn't send in a password, username or password and just got the schedule. And I was like, that's a little weird. Did I somehow like, I don't have cookies set. There's no way that it knows my credentials.
So I visited it in the browser and I was like, Oh, actually that, that inner eye frame there. doesn't require auth at all. And it's a schedule, though, is what I thought. It doesn't really matter. It's just like someone's schedule.
So I push the back button on the browser, or I click the back button in the app, and it takes me back to another page with my social security number and my bank account information. And I realize every single employee's social security number and bank account information is just on the web, unauthenticated. And all you need is this employee ID, a sequential number to find them.
So I try to reach out to the company to tell them, you know, like, hey, this is a problem. Because I knew like my local branch is not going to know anything about this. I try to like reach out to corporate. I like filled out an employee survey and stuff. And they respond back to me. This is a customer support person. My older brother told me, well, you can't give it to them for free.
Like you can't tell them the security vulnerability for free. Like they should pay you for this. Okay. Me being the dumb 16 year old I was. Dubious advice. I was like, well, you know, you'll have to pay me. Didn't hear back for several months.
The plot thickens.
All of a sudden at 6 a.m. I hear a bang on my door. I sleep like on the second story, but like my room kind of goes straight down the stairs to the door. So I'm like the closest to the door of my family. And I hear this like really loud bang on the door at 6 a.m. I go downstairs and I hear police open the door. Like it was like that, like pitch.
And I look outside and there's no police cars, nothing, not a single one that I can see. Cause like we have a big window and I look out, there's nothing out there. And there had been a break-in in a house not too far from us a week prior. And I'm just like, this does not sit right to me. I was like, how do I know it's the police? And the guy responded very confused.
And he goes, well, we're busting in the door. I back up, battering Ram into my door immediately. Throws me on the ground, puts me in handcuffs. And in walks Secret Service agents.
the wildest thing i have a i i don't know if i sure still have the picture but they like left a big huge mark in the front door they had the battering ram ready they didn't even wait very long they just came busting in came busting in uh and so i asked the first secret service agent that is coming in like can i know what this is about as i'm on the ground in handcuffs and he
Next year service agent, which I'll just call a B. I'm not going to give his name, even though I know it. Uh, we'll say secret service agent B goes, what? No, of course he can see the warrant hands me the warrant, which shows the grocery store name. And I know, you know, okay, that's what this is about. I tell the whole story to the Secret Service agent about what happened, what I found.
And I kid you not, he goes, um, well, I think someone overreacted here. But you're facing pending charges of 30 years in two states and at the federal level. Wow. I think my favorite moment from that though, one, so I had this, you know, computer that I, that I had gotten, I still was using it at the time and I like changed out all the parts in it.
And I had this like backpack full of old hard drives. They like tore up my bed, like flipped over the whole room. Like it was absolutely insane. There were eight local police and two secret service agents and they don't take the backpack full of hard drives that was sitting right next to the computer, which is just like top notch police work. Yeah. Yeah.
And then the second was we had gotten an iMac at that point, and I watched as two local police looked around, and they're like, where's the rest of it? And another guy had to be like, I think this is the whole thing. It's just all in one. It was great. So yeah, I didn't do any real hacking. I just found something unauthenticated. It was not complicated. I was not some massive hacker.
I ended up getting interrogated by the company for like eight hours.
They just dismissed the charges or how did it turn out?
Yeah, no charges were ever filed. It was just a search warrant. It was pending charges based on what they found, which of course there was nothing. I didn't do anything. I just found a security vulnerability. But my mom made me tell the company because she was worried I wouldn't have a job or whatever. And I knew they didn't know I worked for them, which was true.
They were a very disorganized company. But I ended up getting interrogated by some lady from corporate who, when I asked for a lawyer to sign documents, she stood up and screamed that I threatened her. And I got fired, not for anything I did there, but for threatening her.
Wow.
Yeah.
So that story plus Miss Johns together got you this job. Because they're like, well, you must be good at what he does. Secret services after him.
Yeah. Yeah. Uh, I don't think she even knew the, you know, like technical details or whatever, but it was enough. They were intrigued that they let me get an interview. And luckily, uh, yeah, I got the interview, I think six weeks into my internship, I got hired on full time and yeah.
Were you actually good?
I mean, I could fake modesty, but yeah, I was good. I mean, I wasn't good compared to now. I was awful. If you look back at all of that stuff, I was terrible. But for where I was at as a junior developer, basically like a recent grad, I knew my stuff. I was able to... When I joined...
We took the backlog from like 60 items in the queue that had been there and I was able to get like 40 of them solved pretty quickly. I mean, it's why I got hired in six weeks from intern to junior developer. Turns out like all the stuff I had been doing in my spare time was actually real software. I just didn't realize it.
that's cool yeah he's the kind of kid who has got a backpack full of hard drives of course he was good at what he did you don't just accidentally end up with a backpack full of hard drives I really didn't and I think I'm sure like part of the reason I want to share this is not like to brag about myself it's like I because I grew up you know like none of my family had gone to college it was a very you know working class family like I had no concept
that this was something. I read a bunch of tech articles. I'd seen all of this stuff. And I knew some people out in California did this. But as a kid growing up in a small town in Indiana, I just thought what I was doing couldn't possibly be the real thing. And so I was two years out of school. I'd never once considered programming as a job, because I just didn't think I was good enough to do it.
Right. Then you find this code base and you realize how many hundreds of thousands and millions of dollars in labor have gone into this monstrosity.
Exactly. Right. And I've realized like, ah, you know, my worst code is, you know, I write bad code. I wrote tons of bad code at that company. Zero question. I mean, they had to scrap a whole project that I did. Like I made all sorts of mistakes, but like you can not be perfect. And yet. contribute quite a bit.
And these people, yeah, they created value for the company despite this code being awful.
Okay, friends, here are the top 10 launches from Supabase's launch week number 12. Read all the details about this launch at supabase.com slash launch week. Okay, here we go. Number 10, Snaplet is now open source. The company Snaplet is shutting down, but their source code is open.
They're releasing three tools under the MIT license for copying data, seeding databases, and taking database snapshots. Number nine, you can use PG Replicate to copy data, full table copies, and CDC from Postgres to any other data system. Today it supports BigQuery, DuckDB, and MotherDuck with more syncs to be added in the future.
Number eight, Vect2PG, a new CLI utility for migrating data for vector databases to SuperBase or any Postgres instance with PG Vector. You could use it today with Pinecone and QDrant. More will be added in the future. Number seven, the official Supabase extension for VS Code and GitHub Copilot is here. And it's here to make your development with Supabase and VS Code even more delightful.
Number six, official Python support is here. As Supabase has grown, the AI and ML community have just blown up Supabase, and many of these folks are Pythonistas. So Python support expands. Number five, they released log drains so you can export logs generated by your Supabase products to external destinations like Datadog or custom endpoints.
Number four, authorization for real-time broadcast and presence is now public beta. You can now convert a real-time channel into an authorized channel using RLS policies in two steps. Number three, bring your own Auth0, Cognito, or Firebase. This is actually a few different announcements.
Support for third-party auth providers, phone-based multi-factor authentication, that's SMS and WhatsApp, and new auth hooks for SMS and email. Number two, build Postgres wrappers with Wasm. They released support for Wasm, WebAssembly, Foreign Data Wrapper. With this feature, anyone can create an FDW and share it with the Superbase community.
You can build Postgres interfaces to anything on the internet. And number one, Postgres.new. Yes, Postgres.new is an in-browser Postgres with an AI interface. With Postgres.new, you can instantly spin up an unlimited number of Postgres databases that run directly in your browser and soon deploy them to S3. Okay, one more thing. There is now an entire book written about Supabase.
David Loren spent a year working on this book, and it's awesome. Level up your Supabase skills and support David and purchase the book. Links are in the show notes. That's it. Superbase launch week number 12 was massive. So much to cover. I hope you enjoyed it.
Go to superbase.com slash launch week to get all the details on this launch or go to superbase.com slash changelogpod for one month of Superbase Pro for free. That's S-U-P-A-B-A-S-E dot com slash changelogpod.
You didn't introduce the calendar, did you?
No. Yeah. So this is there was a calendar table, which was literally a hand filled in calendar that my understanding at best was that when contractors logged in versus when employees logged in, there were certain days that employees were allowed to log in. But contractors like on weekends and holidays could not log in. It was supposed to be completely forbidden.
And this was the system that was doing it. And so one time, like they filled it in for a few years and it actually ran out. And they had to scramble to figure out how to log in and then just had an intern fill in another five years. Had no idea which, there were so many services, so many programs running in the background, so many cron jobs. They had no idea which one had been locking people out.
I love it.
So just filled it in more? Just one row per day and you just got to fill it out. And it's going to check against that row for the day. And if there's no row, then they can't log in.
Exactly. I also had a task that I don't mention in the article where there was this like 5,000 line Pascal program. that had been apparently like very mission critical for the last eight years. And all of a sudden had started failing and they had no idea why. And I was told to rewrite it in C sharp because they didn't want to support Pascal anymore. I looked at this thing.
It was just, it was only 5,000 lines because it was copied and pasted. Like they unrolled a loop by just like copying and pasting code over and over again. So my end code was like 200 lines and I get it. I make sure that like everything I'm seeing is like identical. I was like really thorough to make sure I got the program right because apparently it was really critical.
I was given like a week deadline, you know, okay. I get it. We put it into, we put it in service. And what it was doing was like sending a bunch of emails about some process or other. I didn't, you know, didn't know the details, but reading from a table, sending emails based on reading from the table, not complicated. And, uh,
I immediately, like the day I turn it on, my manager comes over to me and says, please turn that program off. We are getting constant emails from people. Apparently, this program hadn't run in eight years. It was about something that was eight years old and it was spamming people's emails every half hour because it couldn't find the data it was expecting.
Wow. So it had been, it was defunct basically.
It had been not running for, it had been running incorrectly for eight years. They just finally turned on alerting and that's why they started noticing that it was failing.
Oh, okay.
Yeah.
So you did implement the correct behavior.
I've implemented it exactly right. It's just that, uh, yeah, nobody knew that it was failing because that program that, you know, part of the business had been gone for eight years, but those people were still there. Uh, so yeah.
Who knows, man.
Yep. Well, that's wild. Yeah. I think the other one that I loved was like There were whole programs in source control that were just decompiled sources. So they were C sharp, but they had lost the source code to them. So they just took the binary, ran them through a decompiler and checked them into source control.
Okay.
And you had to make changes to this application. And I don't know if you've ever worked with like C sharp decompilation or I imagine it's unreadable, right? It is completely, completely unreadable. It is compiler output. It's like if you had to work on JavaScript minified code. And I had a person, one of these was our time tracking system.
The way we tracked all employees was a custom built application, but we had lost the source control. We had lost the source code. So it was decompiled. And there was some list that was wrong according to a business person. So I go in there and I'm looking at this like Boolean logic that's been like optimized by a compiler. And I finally like get it.
I write down like the logical formula and I plug it into Wolfram Alpha and it tells me like the simplified form. Oh, cool. It was great. I was so proud of myself for thinking of doing that because it was really complicated. It turns out to be like this and that or this and that or this and that. That was like the whole entire thing. figured out what each of these variables came up with.
I was so proud of myself. I go to the business person. I'm like, all right, here's all the variables. Here's the Boolean logic. You know, what does it need to be instead? And she's like, oh, I don't know. I don't know what any of those variables mean. I don't know what any of these terms mean.
Instead, I had to go change a variable, print out a new list on a piece of paper, bring it to her desk, and she would mark which ones were right or wrong. Oh, wow.
Guess and check.
And then I would try to figure out based on her marks, what the bullet, and we, we did it. It took like, you know, 10 rounds of printing out pieces of paper and letting her mark on them. But, uh, yeah.
And then did you decompile that and throw it in source control or like, how, how did you, how did you recover from the circumstances or did you just perpetuate it?
Yeah. I mean, there was no, you just now edited that. So that was the source code, right? That's the only thing we have is this crazy decompile thing. I cleaned up that little bit there. Right. Made it a little cleaner and added some human readable terms to it. But I mean, there's no way you can fix. It was probably a 30, 40,000 line application. Right.
No way you're going to rewrite all of that in the time given.
why did you stay here? Why'd you keep doing this job? Was it just like a, a weird conundrum slash challenge? Like to like, this can, it cannot be real. And I must stay to see what happens.
I mean, you got to realize this. I mean, I didn't stay there super long, but that was, I probably would have, but I met my now wife and moved, but like the, I mean, there were tons of people who'd been there five, six, eight years. This was one, a small town and, with a big city near it, but there's not a ton of programming jobs, and all of them are kind of equally crazy.
I knew people who had worked at other companies now and come here, and I think this kind of stuff exists a lot more. I mean, there's tons of comments being like, I thought, just like we had here, I thought I worked on this code base. This sounds very familiar. But also, I didn't know any better. I just assumed this is what...
code looks like actually in the real world you know i'd seen some open source stuff but never really dived into it but i'm like oh maybe open source is better but a company this is just what you have to deal with and like while it's definitely the the most story worthy code base i've worked in like all of the things that were bad were just so obviously bad it was not a bad place to work
I would actually rank it on one of the better places I've worked. Not the best, but one of the better places I've worked. Part of that was definitely my position in there. I had a great manager. He was just so supportive, so nice, always made sure that I got the growth opportunities that I needed to become a better developer. He saw...
the potential that I could do and made sure to like help me and get people, you know, get more senior developers to help me learn stuff and challenge me. That was really good.
But also like, I mean, this will show my very strong bias here that I know you all might not necessarily agree with, but there was no like agile process or none of that stuff, which I have found to be like the main factor in job dissatisfaction for myself. So like the lack of process was so freeing and so nice. Yeah.
Well, you're not going to get a disagreement from me on that one, but maybe.
We don't do agile around here. We do whatever we want, basically.
Yeah, I wasn't sure. You know, I don't know your exact thing, but like with the Kaizen and all of that, you know, obviously.
Well, the Kaizen just means continuous improvement. You know, that's something that I'm sure you're into, right? Like, let's make things better all the time.
Yeah, I do like, I mean, your Kaizen episodes are great. You know, just wasn't sure. I know it's always contentious when I say, like, not a fan of Agile, even with a lowercase a, just not a fan. So it was a good company to work for. You know, the biggest problem, really, why I wouldn't have stayed there longer is, like, being underpaid. You're in a small town. There's limited talent.
There's limited places that you can go, though, so it can pay you way less. Turns out I was making the same as people who had been there eight years and who had job titles way over me, which was just wild. But this is programming in the Midwest. I know tons of people who are at companies like this with equally crazy code.
I mean, even Salesforce has a branch here in Indianapolis and all the code I hear out of Salesforce sounds...
similarly wild to this when you have so many people and so much I guess momentum you have to make progress and sometimes progress is like just duct tape that part you know and that's kind of what like the employees table this seems like duct tape like why in the world do you have to drop this table
at 7 15 every morning and then repopulate it with a new injection and then people can't log in like why is that the way or the same thing with the sales numbers like why do they have to like claim these wins and then put it on this board and they were able to subject these interns to doing all this minion work basically to get their seemingly their numbers projected properly i don't know like it's it's hard to decipher exactly what's happening there but
Yeah, it's a little complicated. It is. The basic idea for that one, just to be clear, is like salespeople, you know, obviously there's like the real accounting. I know there were some comments on Hacker News of like, that just sounds like fraud. There's like the real accounting system. And then there was the reward system where they get bonuses and stuff. This was the reward system.
And, you know, yeah, they were trying to get bonuses every month. And if they made a big sale at the end of the month and they had already gotten their max bonus, they would just move that to the next month so they could get their bonus for next month. Right. So they had a cap on commissions and things like that. And this was moving it around.
That's funny because I was encouraged when I was in sales back in the day. Like, hey, let's just move that sale to next month because you've already reached your quota this month. Like, good for you.
Yeah, and it got encouraged here, but the way that it was implemented was interns manually writing SQL statements.
Take that hourly wage out of your bonus. It costs a little extra. There's some intangible cost there on that bonus system.
There were people who the whole internship they were there, they never got to write any code other than these SQL updates. And I think that's a shame, right? Like that's not how you should treat your interns. But that was one of the things like, I just refused to do it. I just never wrote one.
So, you know, every time it would be assigned to me, I would just not go find something else to work on and do that instead. I wasn't a great employee also, to be clear. I was 20. I was very, you know, I was very, a little bit more prideful than I should have been. A little bit more arrogant than I should have been for sure. But yeah, no.
What about these hard drives and gilfoil? Is this a Silicon Valley nod or is this a real gilfoil? It can't be really a gilfoil, right?
No, no, it's definitely a Silicon Valley reference.
Okay, good.
Yes, yes. I did not do it as bait for this show, but I guess in hindsight I should have thought of that as a benefit too.
It definitely was like, when I saw that, I thought, Adam's going to love this. There's someone named Guilfoyle?
When you say it, let's call him Guilfoyle. So I figured, and then I immediately Command-F'd and typed in S-I-L and found nothing for Silicon Valley. I thought maybe at least reference it, like, let's, you know.
No, I figured it, for anyone who knows, you know, is a good reference and anyone who doesn't, it's a good enough name, right? It's kind of a fitting name, I feel like.
Sure.
So yeah, no, his actual name, I never, I wouldn't, I felt a little awkward like using his actual name because I never met him. I have no idea who this guy was other than through his code. And so, yeah, he was, he now, I don't think, I looked him up. I don't think he's a programmer anymore. He sells exotic pets.
So yeah, I thought Gilfoyle just seemed like a fitting name, especially if you know the reference, right? It was, that was always the sense that I got of this guy. And yeah, he was, I mean, the most prolific programmer I have ever seen. The amount, the sheer amount of code in that code base, that was his.
And the sheer amount of like applications that random customers or customer support people would have that were just from scratch Windows applications that he wrote with like complicated logic. He would anytime because he refused to use source control, which was why we had his hard drives rated on Munch's desk. He refused to use source control.
If a person asked for a code change, a feature change, he would just rewrite the application from scratch. And he was apparently from everything I heard that fast that like, like pumping out a brand new, like 10,000 line application in a day was like not unheard of for him.
But the code is like a legend. This is like a legend. Yeah. Total legend.
Yeah, absolutely. And like, yeah, I don't, because he didn't use source control. I have no idea how fast he actually wrote this code. Right. Like, so you don't get any history on it, but like, that's how the legend continues.
You know, he can't have a trail, no paper trails.
Yeah, I wrote this in a day. Maybe he just anticipates needs way in advance, right? Or puts his code through some obfuscation every single time so it looks like it's slightly different, right? I don't know what it was. But yeah, his code was... It was a trip to try to understand, though. There was just every time... There was never a consistent pattern to the craziness.
It was like he just woke up every day and thought, what new weird programming pattern can I...
abuse here to write this application services that were pure functions that like i literally do not understand why they existed clients that were just like these these super thick clients that like i mentioned the article completely empty classes classes i i did not exaggerate in the article they were empty classes they had a class definition method definitions
But there was no code in any of the methods. And it was all to build up a structure. And the hierarchy would go 10 layers deep of inheritance. And it was all to build up the structure that then would become a pipe delimited string sent over a socket.
What?
That was all driven by the database. And so when you looked at it, you were like, OK, what does even happen? Once you even figured out, oh, this is a data structure, not a class hierarchy. It was like, wait, what do they turn into? Oh, well, it's like dynamically choosing which column of which table to go grab this field from. And now it's custom there.
And then the table would encode like, how do you encode the value? So like you could infinitely configure how the delimited string would be created. But like, there was no reason to do that because it's an old application that hasn't changed in 15 years. And the same message was written every time it was, it was wild stuff, but I actually debugged that bug for a year.
It manifested itself as like 15 different cases in the system where things would just go wrong. And I thought it was like a memory leak for the longest time. I thought it was all these things. And like finding out that it was just some legacy third party application reused unique IDs every month was like the most exciting and most let down bug find I've ever seen in my life. Yeah, not exotic.
I thought for sure it had to be Gilfoyle's fault, right? His code was too clever. I really wanted to blame him on his cleverness.
I was going to say, it's convenient sometimes to have a Gilfoyle. It's like a patsy. When something's wrong, you got someone to place the blame on because he's been prolific and he's done all these things and he's not around. So surely, gosh, Gilfoyle, what's wrong with you? But no, this was a third-party system.
What about ops in this case? Like you're talking about the code base, but somebody's got to keep that database up and it seems like it's getting hit in like wild ways. Like this chain function, for example, it's probably got the database spinning, the disks are spinning. And I'm assuming that's the day of hard drives.
Yeah. So the, the database was definitely, we had quite a few DBAs, like the DBA to code to programmer ratio was pretty high. And they, we had some very beefy machines running this SQL server set up.
On-prem? Yeah.
On-prem.
Obviously.
So yeah, we had everything. Actually, at the office I was, there was a data center area as well as a shipping area. So it was on-prem, it was local. There was some talk about the company building a private cloud system and things like that, but it's a little early for them to actually do that. Nothing ever came of it. Yeah, there was definitely some beefy things. Most of ops, though...
Really ran through one guy who was really good at his job. He was the ops guy. I mean, there were technically other ones, but everything I ever dealt with, it was him doing it and maybe delegating some tasks occasionally to other people. But it was a pretty bespoke kind of setup. Deployments were all done by hand by him late at night, so that way it wouldn't affect anybody.
It was that kind of place, that kind of setup where servers had pet names and all of that. It was well before. There was some early maybe looking at Puppet, maybe doing some of that, but... nothing really came of it.
So yeah, it was a, it was a pretty, this is, this is honestly one of the things that like was so strange is it's the scale of the actual code base, the scale of like how many people are using this is small, but not completely trivial. And yet like this, this legacy code base, especially the other ones, the other applications that were like the big rewrites had not seen production use and
And we were able to, with me, an analyst, a QA, one senior developer, and four interns, we're able to out-compete all of this rewrite. We were adding new features, fixing old bugs, and doing all of that in this system. While they were off doing their big agile processes and doing all their story pointing and all of that and never getting anything done. So it was fun.
When you actually think about it, if you're not one of these big web scale companies, servers are simple. Code's not that complicated, even if the code's complicated. You can make changes if you just don't get in your own way. So, yeah.
It sounds fun. I mean, I would like to work on this code base for maybe like a month and then move on. But I'd like to visit.
As a game, maybe.
Yeah, it is a game. I mean, it's got to feel like that to a certain degree.
It did. It felt very much like a game. But I think that's how I approach most of this work. Like I said, I think the job I'm at now, we've got a big, massive old code base. I now work on a fork of Rhino.js.
Okay. Yeah.
I don't know if you all remember Rhino.
I do, but I can't remember what it does. But I remember Rhino.js. It was involved in my cappuccino days. I think they were using it back with Objective-J.
Yep, that sounds about right. It is a JavaScript implementation written in Java. So there's a compile to JVM bytecode and an interpreter all written in Java. It was abandoned years and years ago, but I now work at a company that has a long-term fork of it that runs millions and millions of lines of customer JavaScript.
Really?
And it's got some custom features.
Say more.
Sounds good. Yeah, so we're now trying to refactor away from it. But for example, in our original version of JavaScript that people still use to this day, if you did dots, it was like doing question mark dot. So if it was undefined, it wouldn't throw an error. It would just return undefined. This was a choice by the founder of the company.
Like, I guess I just have a knack for finding these code bases. I don't know what it is. Where things are just crazy. And I think a lot of developers work in these kinds of things, right? They just don't talk about them publicly. Totally, totally.
I've had similar experiences, all I think at smaller scales, both in terms of company size and code base. My craziest one was I inherited, I did a rescue project for a boat shop somewhere in Georgia where they just needed, like basically it's a Ruby on Rails application that ran back office for a boat shop. A lot of merchants, a lot of sales, a lot of this kind of stuff.
So similar tables and stuff, which is why it's probably resonating with me. And they had lost the original dev team. It was like a contract team came in, built this system and left. And then the IT guys who are also third party kind of took the system over because they were just nice guys who were helping out the company.
That was a successful boat retail shop and relied upon this application to run their business now. And this was like really early Ruby on Rails days. I think it was like version 1.2 or something. And so there's a lot missing. And this team came in and wrote a lot of very clever code. Basically implemented a meta framework on top of it. And it was insanity. It took me a very long time to unravel.
And then they left, you know. And these people were left kind of high and dry. And so I was happy to help out. And I had the challenge, the game, like all these things. There was no... development system. It was like production and I copy the code down and try to get it running on my machine, you know?
So you're very much like doing crazy stuff, very small increments at a time, try not to break things. Um,
And so that experience just resonates with everything you're saying, like the metaprogramming, specifically when you're talking about the thing that generates data structures from classes, and the database kind of is the programming system as well, if you want it to be, but no one's using it, like all that stuff was there.
And it took me a very long time to be able to unravel it and understand it to the point where I was like, oh, it's kind of clever once you know how it works, but... That's a terrible thing. And so I think, and I didn't write up about that at all, I think there's tons of code bases like this out there.
Yeah, I wish I had finished this write-up. I haven't actually done much Ruby, although I did work at Shopify on YJIT, the Ruby JIT compiler, for a little bit before getting laid off, sadly. It happens. But the one Ruby code base I worked in, there was just this little bit of super clever metaprogramming, which, of course, Ruby loves. Lots of people in Ruby love their metaprogramming.
And I could never find, the reason I never wrote it up is I could never find a non-circular starting point. Every time I would want to explain something, I would have to, because the code meta-looped on itself every single time. There you go. See, that's the problem.
It's like a time travel movie, which we talked about pre-show, so not a good callback. But yeah, that's one of the problems with meta-programming is it's just too meta sometimes and you can't unravel it.
I think this is, not to self-promote or whatever, but this is one of the reasons that I really enjoy the Future of Coding podcast. Because I think there's a lot of code out there that people aren't happy with, and I don't think we talk about it. In some ways, I think looking at this legacy code base is the same thing for me as looking at the shiny new stuff.
I think oftentimes we don't appreciate code for what it is. We don't look at what's come before. We kind of look at legacy systems like they're all bad, and there's nothing good to gain from them. I've seen a bunch of blog posts talking about crazy code, but one of the things I wanted to do in this article was talk about crazy code, but in an endearing way. I loved this crazy code.
I just love code. I think no matter how bad it is, it's one of those things that... Code is a medium to put information down that like is unlike writing. I, you know, you, you can get a sense from this application, like from my blog post, what writing code in this code base was like, but like you said, like go and do it for a month. It's fun. It's interesting.
And I think there's so much more code out there and so many different ways of writing code that we just haven't really unlocked yet that I, you know, I want to see us do more of.
Yeah.
Well, our friends over at Speakeasy have the complete platform for API developer experience. They can generate SDKs, Terraform providers, API testing, docs, and more. And they just released a new version of their Python SDK generation that's optimized for anyone building an AI API.
Every Python SDK comes with Pydantic models for request and response objects and HTTPX client for async and synchronous method calls and support for server sent events as well. Speakeasy is everything you need to give your Python users an amazing experience integrating with your API. Learn more at speakeasy.com slash Python. Again, speakeasy.com slash Python.
And I'm also here with Todd Kaufman, CEO of Test Double, testdouble.com. You may know Test Double from our good friend, Justin Searles. So Todd, on your homepage, I see an awesome quote from Eileen, you could tell. She says, quote, hot take, just have Test Double build all your stuff, end quote.
We did not pay Eileen for that quote, to be clear, but we do very much appreciate her sharing it. Yeah, we had the great fortune to work with Eileen and Aaron Patterson on the upgrade of GitHub's Ruby Rails framework. And that's a relatively complex problem. It's a very large system. There's a lot of engineers actively working on it at the same time that we were performing that upgrade.
Being able to collaborate with them, achieve the outcome of getting them upgraded to the latest and greatest Ruby on Rails that has all of the security patches and everything that you would expect of the more modern versions of the framework, while still not holding their business back from delivering features, we felt was a pretty significant accomplishment.
And it's great to work with someone like Eileen and Aaron Because we obviously learned a lot. We were able to collaborate effectively with them. But to hear that they were delighted by the outcome as well is very humbling for sure.
Take me one layer deeper on this engagement. How many folks did you apply to this engagement? What was the objective? What did you do, etc. ?
Yeah, I think we had between two and four people at any phase of the engagement. So we tend to run with relatively small teams. We do believe smaller teams tend to be more efficient and more productive. So wherever possible, we try to get by with as few people as we can. With this project, we were working directly with members from GitHub as well.
So there were full-time staff on GitHub who were collaborating with us day in, day out on the project. This was a fairly clear set of expectations. We wanted to get to Rails, I believe, 5.2 at the time and Ruby 2.5. Don't hold me to those numbers, but we had clear expectations at the outset.
So from there, it was just a matter of figuring out the process that we were going to pursue to get these upgrades done without having a sizable impact on their team. A lot of the consultants on the project had some experience doing Rails upgrades, maybe not at that scale at that point.
But it was really exciting because we were able to kind of develop a process that we think is very consistent in allowing Rails upgrades to be done without like providing a lot of risk to the client. So there's not a fear that, hey, we've missed something or, you know, this thing's going to fall over under scale.
We do it very incrementally so that the team can, like I said, keep working on feature delivery without being impacted. but also so that we are very certain that we've covered all the bases and really got the system to a state where it's functionally equivalent to the last version, just on a newer version of Rails and Ruby.
Very cool, Todd. I love it. Find out more about Test Double's software investment problem solvers at testdouble.com. That's testdouble.com, T-E-S-T-D-O-U-B-L-E.com.
It's very easy, especially when you come into a code base to, you know, Guilfoyle the thing in the sense of how I was talking about Guilfoyle earlier, where it's like some other who was dumb or incompetent or malicious did this. But as you actually start to like work with it and talk to people and learn about it, okay, there are politics and there are power struggles and stuff.
There are things that happen because we're humans, but a lot of it is like, They made the best decision they had at the time with the information that they had. And I'm staring at it with completely different perspective years later. And like that stuff is fun to learn. And you actually realize like, yeah, that duct tape was really reasonable considering all the things they considered.
I just don't know those things. And so I really do appreciate code from that perspective, especially legacy systems that are still powering businesses and bringing value to people is that we want to be smarter than everybody else, but. Those people just had different contexts lots of times that we just don't have.
And you start to learn those contexts and it gives you a new appreciation for the code that you're looking at. And that's cool.
Yeah, Peter Nauer of Backus Nauer Forum, so BNF. Peter Nauer actually has this great paper called Programming as Theory Building. And one of the things in there that I think is really interesting is he talks about code bases dying. And by that, he doesn't mean that the code isn't running anymore or no one makes changes on it.
He means that the people who knew the original context of the code base are no longer there. They're no longer working on it. And so everyone else that is having to make changes to this code base is working on a dead code base that's no longer alive because they've completely lost the theory of the code base. Why is this here? Why put these lines...
in the way that they are, how do I make these kinds of changes? And he argues in there that you can't revitalize a dead code base. Once a code base dies, you can create a new theory about the code base, but it's impossible to ever recover the original one. And I feel like I've worked a lot on dead code bases where the people who knew that context once
are long gone, don't work at the company anymore. And in some ways, this code base wasn't dead because of Munch. Because Munch could always provide that little bit of context, breathe some life back into it. And had Munch not been there, the code base would have been completely dead. No one would have had any idea what was going on.
Is Munch still there?
I haven't checked. Um, I don't know if he even has a LinkedIn or anything. Um, maybe he does, but probably hasn't updated it even if he did. So I'm not sure. I know the company now doesn't have that name anymore. I don't know if that, you know, I don't know if that code base is still running or not. You know, it, it might be, but it also might because it was customer support and sales, um,
When you get a big merger like that, that's one of the things that often gets, like they just choose one, right? Credit card processing code, I'm sure, is still running. But customer support might not be. Code base might literally be dead now. Merchants might be gone. Merchants 3, if they ever got to that point, right, might not have any more data in it. I don't know. Yeah.
I do like the way that you compliment it, though. You talk about the beautiful mess, that section there where you talk about, which is kind of what you're talking about here, where there were no overarching design systems to work in. There was a lot of freedom. There was no documented design system. You mentioned that there was any concerns of co-duplication were out the window.
You can sort of carve out your own little section because trying to fix the big mess was impossible. So you just gave up and just worked in your own little world of insanity, as you say here. I think that's kind of cool that somehow, some way in this mess, you found beauty to enjoy.
I think it's something that we should do more of. I've worked in systems since that are a mess, but everyone's always trying to fix it. And I get that impulse because I hate the mess too. But sometimes it's just beyond fixing. This is why people give the advice of don't do the big rewrite, right? Because it's much bigger than you think it's going to be and it's probably going to fail.
And I think the same could be said of trying to make sense of a 10-year-old, hundreds of thousands of lines codebase. There's just no way you're going to be able to hold it in your head. And when you try to come up with some overarching scheme of what the code base is really doing, and now I can come up with the perfect abstraction and it will do all of this, I just think it's a losing prospect.
It's just not going to work. And so, yeah, I think we should embrace more... Even in good code bases, I think we often... want uniformity. We want everyone to have the same coding style. We want everyone to follow the same coding rules. And I've found that often, to me, that ends up causing more problems in the long run.
Because once you have to have everything consistent, as soon as there's a big change that needs to be done, it actually becomes harder. Because you can't do the big change all at once. And if you require consistency, it's harder to do those small changes each time.
Even the connection to the users is interesting where you put in the after section where you describe it as a ragtag of juniors, essentially. All the serious senior folks have gone away. Even like we mentioned this as a game, it's kind of interesting to think about this as a thought experiment of this ragtag of juniors just figuring it out, talking directly to the users, talking directly to...
The support rep who's got the problem. How can I make your life better? To me, that's like in the grand scheme of things, it seems kind of interesting. Cause I said, Hey, why'd you stay there? And you're like, well, because, and I think I can kind of see that light now in the after section.
Yeah, and the next job I took actually had the exact opposite thing, where we were actually completely forbidden from ever talking to our users. Not just as developers, as a company. I worked at a big company that was, it's a big private company, they buy a bunch of companies. And we were building an application for a sister company in this big group.
And they hired a contractor to be the go-between. And they were scared of us ever talking to end users because they were worried those end users would think that their job was in jeopardy if we were rewriting this application. So we were never allowed to talk to them. And I worked at that company for a little while. And then I left and then went to a startup that didn't do very well.
And I came back as a contractor at the original company. And at that point, they had changed this rule where they now were allowed to talk to their users indirectly. Not directly. How so? Well, okay. So there were actually now users who were in the exact same building, two doors down from where the developers were, but they couldn't go talk to them.
What they could do was those people would write three by five cards of feedback and they would post them on the wall. And I came back as a contractor and I was so interested to see like, you know, what is the feedback on the system that I knew was not a great system. It was... It was a completely greenfield, brand new system.
And because of organizational issues, as you can imagine, it did not meet user needs. And I looked at a few of them. They were all like, you know, please fix this UI element. Please do these things. But one of them will always stand out to me. And it was, very simply put, it was, remember, you're supposed to make our lives better, not worse.
Wow. That one hits hard, doesn't it?
Yeah, yeah. And that's what this whole, that's what I had done. Like, the whole application I had been building with this team of 30 people was making these users' lives worse. That sucks. I mean, that's...
that's not how it's supposed to work. Yeah.
Yeah. And so like, that's what I, you know, looked at like the contrast between this first job where, yeah, things were an absolute mess. Whereas there, I mean, there was no tests, right? Like they just literally didn't exist. Yeah. of any sort, not unit tests, not integration tests, not anything. We had manual QA that sometimes did some things. I had one really good QA for a while.
She was fantastic. But by all standards, it was wrong. Whereas the next job, by all standards, it was supposed to be good. It was right. Yes, everyone has different opinions, but it was Spring, and it was Angular, and we had 100% test coverage. That was the rule. Every line, every if, everything had perfect test coverage. We had an end-to-end test coverage suite.
We had dedicated technical QA that did all of this. And yet it was a way worse code base. It was plagued with constant problems that it didn't meet the company needs. It didn't meet the end user needs. It couldn't scale. Everything about it was wrong, despite like on paper, we followed all the best practices. obviously a hundred percent test coverage is not actually a good metric. I know that.
Well, I did get that one changed to 70 at some point, but I guess what's your broad takeaway from that circumstance? I, I think that one of the things that I've like come back to over and over again in my career looking, and I think this first job really did teach me this is like as programmers, it's very easy to give up on our responsibility and,
And say, I'm just doing what the business wants me to do. And that's what my job is. My job is to do what I'm told. It's to complete this story. Yes, I can maybe sometimes give input. But ultimately, I make it happen. But I don't decide what we do. On the how, not the what.
And I think that that's always been the problem I've seen at these companies when things went poorly, was when developers just kind of gave up on doing what was good and what was right for the system and for their end users and for the code. in order to just do what they're told. And I think that as programmers, we have to accept the fact that we're not hired to do as we're told.
Otherwise, we just wouldn't have the salaries we do. They wouldn't pay us this much just to not want our opinion. And even if they say they don't really want our opinion, We got to do what's right. If you're a good friend with somebody, you don't always do what they ask you to do, but you always do what's right for them. And that's how I think that we have to approach these things.
And every company I've been at where the software made people's lives worse and not better, programmers have kind of given up on doing what was right and just did what they were told.
Well said, Jimmy. Well said. Anything else? Any stone that we've left unturned? I'm sure there'd be dragons elsewhere, but anything you'd like to highlight before we call it a show?
No, I mean, yeah, there's tons of other stories. There's tons of other craziness, like the time I had to split the code base in half and all sorts of weird things like that. But they're a little long.
Fair. Well, great best worst blog post. I love that you wrote that up. I think there's a reason it was popular. I think because it's a shared experience, well documented. And it's fun to laugh at these things. We laugh so we won't cry. Or maybe we laugh at you and not with you because I didn't have that particular problem.
But I appreciate you coming on the show and talking to us about it and for giving us some big picture ideas and hope for the future of coding and also for the current state of coding and code itself and a love for it that I share at least, even when I despise it sometimes. I still love it. And I think that that's just the way it is sometimes. Adam? I agree. Yeah.
I agree. All right. You never know when you're the best of times. That's right. Right. This seemed like from the outset, like maybe not the best of times, but actually it kind of was.
Absolutely.
Are you talking about our podcast? Are you talking about his experience? His experience. I know. His experience. No, it was fun, Jimmy. Thanks for sharing. Thanks for going, I think, deeper than maybe is necessary to share a story that may not even matter to anyone else besides you. And you just shared it with the world.
And now you can reflect on some of those key attributes that really reflect back on a good time. Really. It's cool. Thanks. I enjoyed it. Yeah. All right. Bye, y'all. So Jimmy looks back on this time, this beautiful mess, as he said, the best worst code base as one of the greatest times in his life as a programmer.
It's kind of funny how you can look back on times when you're in the moment and things kind of suck or they're not seemingly so awesome. And it's actually one of the best times of your life. It's kind of funny, right? As one wise person has told me before, these are the days. Okay, so that was a fun show with Jimmy. Very fun adventure. Very deep context we got to dive into.
I hope you enjoyed the show today. Make sure you check out Jimmy's podcast, Future of Coding. You can find that at futureofcoding.org. So I want to mention this here in the post show because we're still not quite there yet, but we have definitely dove deep into the Zulip podcast.
rabbit hole if that's even a thing I don't know maybe there's an analogy there I could have came up with but I didn't but we love slack for many years and then recently we tried Zulip instead and so if you're already in slack there's an invite there in the main channel you can take advantage of to come over to Zulip and to test the waters.
Many of the folks who have done so already have said, never go back to Slack, that Zulip is the best. And I think that's how we feel. We've talked about it on a recent episode of Change Logging Friends, Jared and I. But I want to invite you, go to the show notes. There's a link in the show notes for you to join our Zulip. Everyone is welcome. There are no imposters. Hope to see you there.
We had some really awesome sponsors for today's episode. Assembly AI, the leader in speech AI. Check them out, assemblyai.com. Our friends over at Superbase, check them out at superbase.com. Our friends over at Speakeasy, check them out at speakeasy.com. And, of course, our friends over at TestDouble, testdouble.com. Just have TestDouble build all your software, and you'll be good. There you go.
Okay, last but not least, our friends, our partners over at Fly. Yes, that's the home of changelog.com. And that is the public cloud built for developers who ship. Deploy your app in five minutes at fly.io. Okay, those beats by Great Mess of Cylinder were banging. Gotta love BMC. That's it for today's show. We'll see you on Friday.