Gerhard Lazu
Appearances
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Pipe dream one word, pipe space dream, pipe tap dream. I'm not sure about that. I think we can do one like us for fun or camel case indeed.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
I'm leaning towards one word though just like okay just pipe dream one word okay and I'm leaning in the other direction so we need a poll great well the repo name is already like lowercase pipe dream no spaces no nothing no no dashes nothing like that so you know I think it would make sense so yeah all right we'll run a poll See what people think. See what people want.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Give the people what they want. Correct. And when it comes to, when we do switch it into production, whenever that happens, I think we could maybe discuss again, whether we rename it, when it stops being a pipe dream for real. For now, it's still like a repo. It's still a config. It runs. I mean, if you go to pipedream.changelog.com, it does its thing.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
But it's not fully hooked up with everything else that we need. I have a new name.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Pipe media? I don't know. Pipelog? Pipelog. Oh, here's a better one. Change pipe. Pipely. Pipely. That one really hurts. I think that's the winner. I think that's the winner. Quick, buy the debate before someone else buys it. Pipe.ly. Oh, yes. That one's almost too good. Almost.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Well, based on the journey so far, I would say yes. I mean, it would definitely put us in control of the CDN too. A CDN is really important for us. So it's even more important than a database because we're not heavy database users and we'll get to that in this episode, I'm sure. So a CDN really is the bread and butter. Now, we need something really simple.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
We need something that we understand inside out. We need something that I would say is part of our DNA because we're tech focused and we have some great partnerships and we've been on this journey for a while. You know, it's not something that one day we woke up and we said, let's do this. So this has been in the making for a while. We were almost forced. In a way, yes.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
I would say encouraged, you know, in a way, like we're pushed in this direction. There are other options. Yeah. But I think there is like this like natural progression towards this. And it doesn't mean that we'll see it all the way through. But I would say that we are well on our way to the point that I can almost see the finish line. I mean, even the roadmap, right?
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Putting the roadmap down on paper, it made me realize actually the steps aren't that big and we could take them comfortably between Kaizens. And I don't want to say by Christmas, but wouldn't it be a nice gift, a Christmas gift? What do you think?
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
No, there isn't a test harness now.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
I would love that. Okay. Yeah, go for it. Cool. Open source for the win. Cool.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Honestly, I think so. Honestly, I can definitely see this being part of Flutter.io.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Are you thinking make it private? The repo? It's still not too late.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Well, yeah, no one's using it, so... Yeah, private rug. It has 60 lines of varnish. I think we're getting ahead of ourselves, right? I think so. But once we start adding the test harness, once we start adding the purging, which, by the way, is specific to our app, but maybe...
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
that would need to be generic by the way so if we this was to be a product we would need to have a generic way of purging doesn't matter what your app is so there's a couple of things that we need to implement to make this as a product and in that case it would be in this repo i think but um it could also be like a hosted service like tigris is maybe especially if you get the cool domain Why not?
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
I can see that. And this can be our playground, like the pipe dream can be our playground. But then the real thing with all the bells and whistles could be private.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
The one thing which I do want to mention is that there's a few people that helped contribute. So I would like to, this is also time for shout outs. Of course. To Matt Johnson, one of our listeners. Shout out to Matt. And also James A. Rosen, he was there from the beginning. The first recording that we did, that's already live. The second one as well that we recorded, I haven't published it yet.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
I still have to edit it. But that was like basically the second pull request that we got together. And even though a bunch of work, you know, went obviously in the background before we got together, when we did get together, it was basically putting all the pieces, you know, so we did like in this very open source group spirit. And yeah, so there's that.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
So I think keeping that true to open source would be important. And if not, then we would need to make the decision soon enough so we know which direction to take. But you're right, rug pulls, not a fan at all. We should never do that. And even the fact that we're discussing so openly about this, I welcome that. I think it's amazing, this transparency.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
So that we're always straight from the beginning what we're thinking, so that no one feels that they were misled in any way. Agreed. Agreed. I like it. Well, the last thing that I would like to mention on this topic before I'll be ready to move on is that we live stream the CDN journey, a change log with Peter Bandugo. There'll be a link in the show notes.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
We got together and we talked about where we started, you know, how we got to the idea of the pipe dream and where we think of going. So if you haven't watched that yet, it'd be worth. there was a slideshow. Not as good as the last one, the last Kaizen, but it was, I'm happy with it. Let me put it that way. Awesome. Cool. We'll link that up.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Custom feeds. That's one of your topics, Jared. Custom feeds. So tell me about it. I don't know what it is. I know what it is, but I don't know what exactly about custom feeds you want to dig into.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Well, I share the slideshow. Okay. I mentioned last episode, I have a slideshow with my talking points, couple of screenshots, things like that. This time I shared it ahead of time and I prepared ahead of time as well. But also I've been making small updates to the discussion, I think more regularly than I normally do. Discussion 520 on GitHub.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
To build one now. Oh, you would? Yeah. Live on the air. Let's see what happens if we do that. So changelog.com. How do I do that? Like run me through that, Jared. I sign in.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
get started. Okay. New feed.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Hang on. I'm still a tagline. Jared made me do this. Okay. Okay. Moving on.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
okay so hang on i used oh i see okay okay i see i see so upload cover art that's the thing which was messing with me because i wanted to add mine but then it said or use hours and when you say or use hours i'm basically changing the cover art which i uploaded
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Okay, so I've made a few changes. First of all, the name is Gerhard and Friends. Okay. The description is Kaizen 16, this episode.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
The cover art, I uploaded one, but then I changed the changelog and friends. Okay. Starts today, 13th of September. Yes. Title format, I will leave it empty. And for the podcast, I'll choose changelog and friends. Okay. Yes. And this feed should contain ChangeLog++ at free extended audio. Yes.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
And automatically add new podcasts with launch. I'm going to deselect that because I only want to change with my friends. Save.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
And I found the first bug. No, you didn't. So the bug is, if I upload my cover art and then I select another cover art from one of yours, it uses my cover art, but not in the admin. In the admin, it shows me that it's using yours, but when I create the feed, it's using my cover art. Okay, so you did both.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
I mean, we always have one for every Kaizen, but this time I just, you know, went a little bit further with it. And I think it will work well. Let's see.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
I will. Let me take a screenshot so that I remember. Boom. There. Awesome. Cool. Looks great. Actually, hang on. The picture which I want for this cover art is us three recording right now. So if Adam looks up, I'll take a screenshot. There you go. That will be my cover art. Okay. Got it. So good.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
including mine including yours I see yours right there amazing and the cover is your face correct yes cool so cool so that's the feature that's amazing it worked very well I have to say I just still have to load it in my podcast player but once I do that amazing well let's stop there then because that's where I'm at and that's where I'm stuck Jared you're also stuck yes so Gerhard's next step is to do what I've done and I think he may have the same outcome I don't know
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Hey, Adam. Everything's up. Whenever someone asks me that, everything's up. That's the SRE answer. Everything's up. Everything is up. Otherwise, I'm not here. If something's down, I'm not here.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
So yours has a bunch. The URL only contains feeds. It's forward slash feeds, forward slash Asha. So it doesn't have the full.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
And you've been waiting for this to do it live? Why would you wait this long?
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Now, interestingly, if I hover over it, I can see the correct link.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
But when I click on it, I only get the path.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
So I have a quick fix. You right-click the get URL, and you say copy URL rather than relying on the click action. And then you get the proper URL. Try that, Adam. Let's see if that works.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Yep, so everything's up. I like that. Well, last Kaizen, we talked towards the end about the pipe dream. Oh, yeah. That was the grand finale. So maybe this time around, we start with that. We start with a pipe dream. We start with what is new. Start where we left off. Exactly. Love it. So we mentioned that, or at least you mentioned, Gerhard, that, was it Adam? Can't remember.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Anyways, we will clarify this after I mention what I have to say. Wouldn't it be nice if we had a repository for the pipe dream self-contained separate from the application? Whose idea was it?
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Zulipchat.com. I'm doing it now. Let's see. Log in. Okay. Log in with Google. Go. There you go. Yes, continue. Okay, sign up. You need an invitation to join this organization.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
No way. There's a Kaizen channel. This makes me so happy.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Oh, wow. Okay. I'm definitely going to check this out. This is nice. This is a very nice surprise. It was worth joining just for this.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Big thumbs up. So amazing.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
That's right. So github.com forward slash the changelog forward slash pipe dream is a thing. It even has a first PR. that was adding dynamic backends. And we put it close to the origin, a couple of things so you can go and check it out, PR1. And what do you think about it? Is the repo what you thought it would be?
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Well, I'm going to talk about one very quick improvement, actually two, which I've noticed. The news. Yes. I love the latest. Oh, you like that? That graphic is so cool. I really like the small tweaks. Also, the separators, the dividers between the various news items. They just stand out more. I really, really like it. And it feels like more... The play button is amazing, by the way. I love it.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
I can definitely see it. I made the play button stand out. Yeah. It feels so polished. Thank you. It really does. But the latest is so amazing. And the news archive, it's there. And it works.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
So I appreciate it. It gives us another dimension. It's playful. It's, you know, like there is some fun to be had here. It's not just all serious. It's not like another news channel, but it's really, really nice. Like the whole thing, it feels so much more polished compared to last time. I can definitely see like the tiny, tiny improvements. Yeah. Very cool. So much Kaizen. Indeed. Cool.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Well, the next big item on my list is to talk about twice, 2x faster time to deploy. This is something we just spent a bit of time on. I was surprised, by the way, of the latest deploy. It was slower than 2x, but we can get there. Okay. The first thing which I would like to ask is how do you feel about our application deploys in general? Like, Does it feel slow? Does it feel fast?
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Does it feel okay? Do you feel surprised by something? How do application deploys when you push a change to a repo feel to you? Historically or after this change?
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Okay, okay. So what would make them not too slow? Is there like a duration? Maybe like a 2x. 2x, okay. That's so leading though. I literally meant like how many minutes or seconds, I think we talked about that. Would it feel that it's good enough?
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Okay. That's a good one. So what used to happen before this change, we used to run a dagger engine on the fly so that it would cache previous operations. Okay. so that subsequent runs will be much quicker, especially when nothing changes or very little changes.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
The problem with that approach was that from GitHub Actions, you had to open a WireGuard tunnel into FLY so that you'd have that connectivity to the engine. And what would happen quite often is that tunnel, for whatever reason, would maybe be established, but you couldn't connect to the instance correctly, and you would only find that out a minute or two within the run.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
And then what used to happen, you would fall back to GitHub, which is much slower because there's no caching, there's no previous state, and the runners themselves, because they're free, they are slower. Two CPUs and seven gig, which means that you have to, when you have to recompile the application from scratch, it can easily take seven, eight, 10 minutes.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
And that's what would lead to those really slow deploys. So what we did between the Kaizens, since the last Kaizen, Let me see which pull request was that. It was pull request 522. So you can go and check it out to see what that looks like.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Well, I think the person whose idea it was should do that. However, I can start. So the idea of the pipe stream was to try and build our own CDN, how we would do it. Single purpose, single tenant, running on fly.io. It's running Varnish Cache, the open source variant. And we just needed like the simplest CDN that we needed. which is, I think, less than 10% of what our current CDN provides.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
So when everything would work perfectly, when the operations would be cached, you could get a new deploy within four minutes, between four and five minutes thereabouts. And with this change, what I was aiming for is to do two minutes or less.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
And when I captured, when I ran this, like the initial tests and so on and so forth, we could see that while the first deploy would be slightly slower, because, you know, there was nothing, subsequent deploys would take about two minutes. Two minutes and 15 seconds, the one which I have right here, which is a screenshot on that pull request 522. So how did we accomplish this?
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
We're using namespace.so, which they provide faster GitHub actions runners, basically faster builds. And we run the engine there. And when... a run starts, we basically restore everything from cache, the namespace cache, which is much, much faster. And we can see up there, basically, per run, we can see how much CPU is being used. We can see how much memory.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Again, these are all screenshots on that pull request. And while the first run, obviously, you use quite a bit of CPU because you have to compile all the Elixir into bytecode and all of that, subsequent runs are much, much quicker. And the other thing which I did, I split the, let's see, is it here? It's not actually here. We need to go to Honeycomb to see that.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
So I'm going to Honeycomb to look at that. I've split the build time, basically the build, test, and publish from the deploy time because something really interesting is happening there. So let's take, for example, before this change, let's take Dagger on Fly, one of the blue ones, and have a look at the trace. So we have this previous run which actually took 4 minutes and 21 seconds.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
and all of it is like all together it took basically three minutes there's like some time to start the engine to start the machine whatever whatever all in all four minutes and 20 seconds so a newer run for example this one which was fairly fast it was two minutes and a half if we look at the trace we can see that diagram namespace the build test and publish was 54 seconds
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
So in 54 seconds, we went from just getting the code to getting the final artifact, which is a container image that we ship into production. In this case, we basically publish it to GHCR.io. And then the deploy starts. And the deploy took one minute and 16 seconds. So we can see that, you know, like with this split is very clear where the time is spent.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
And while the build time and the publish time is fairly fast, I mean, less than a minute in this case, the deploy takes a while because we do blue-green, new machines are being promoted, the application has to start, it has to do the health checks. So there's quite a few things which happen behind the scenes that, you know, if you look at it as like one unit, it's difficult to understand.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
So this was ideal case. This is what I thought would happen. Of course, the last deploys, if I'm just going to filter these dagger on namespace. By the way, we are in Honeycomb. We send all the traces and all the build traces from GitHub Actions to Honeycomb. And you can see how we do that integration in our repo. You can see that we had this one, 2.77 minutes, which is roughly 2.40.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
But the next one was surprising, which took nearly five minutes. And if I look at this trace, this was, again, nothing changed. Stuff had to be recompiled. But in this case, the build, test, and publish took nearly three minutes, which this tells me there is some variability into the various runs when it builds it. I don't know why this happens, but I would like to follow up on that.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
As a TLDR, this change meant that we have less moving parts. And when namespace works, and this is something, again, that we need to understand, why did this run take longer? It should take within two minutes. We should be out. A change should be out in production. Half the time is spent in build, and half the time is spent on deploys.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
So when it comes to optimizing something, now we get to choose which side do we optimize. And I think build, test, and publish is fairly fast. The slower part is the actual deployment. So how can we maybe half that? How can we get those changes once they're finished and everything is bundled? How could we get it out quicker?
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
i love it i think do you have ideas on that well i think the application boot time could be improved right because it takes a while for the app to boot when i say it takes a while it may take 20 30 seconds for it to be healthy all the connections to be established now i'm not sure exactly which parts of those you know would be the easiest one to optimize
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
But I think the application going from the deploy starting and the deploy finishing, taking a minute and a half is a bit long. So I'll need to dig deeper. Like, is it when it comes to connecting to the database? Is it just the application itself being healthy? Like which part needs to be optimized? But again, we're talking, this is like a minute and a half.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
We're optimizing a minute and a half just to put this into perspective. And that's why I started with the question, like how fast is fast enough?
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
And the rest is just most of the time in the way. And it complicates things and it makes things a bit more difficult for the simple tasks. How the idea started, I would only quote you again, Jared. Would you like me to quote you again? That was Kaizen 15.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
No, that's a good one. That's exactly what, you know, we're like trying to hash it out so that we share the understanding that each of us holds. so that we can, you know, talk about like what would, because we talked about this in the past and I really liked Jared's question.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
He was asking, we're talking about like Kaizen Inc and, you know, we're talking about all this change, but are we actually improving? And that's why when I tried to think about this and I was thinking about like, okay, what would the improvement look like? And can we, I mean, we can measure it and we can check, have we delivered on this?
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
And until like the last deploy that went out, I was fairly happy with the time that the duration that these deploys were taking. But based on the one which I have right in front of me, the build going from one minute and a bit to almost three minutes, I think that variability is something that I would like to understand first before optimizing the boot time.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Well, let's open up namespace. Let's go to instances. We can see the last build, which you can see here, like all the builds. This is inside Dagger, is that right? This is namespace. All this is namespace, by the way. So we're using namespace for the runners. And I would like... This is a third-party service? It is a third-party service, yes. That you just found or someone told you about?
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Exactly, yes. I am paying attention to various build services and depot.dev. I love it. Namespace.so. Namespace.so, yes. Our trial is almost over. Exactly, yes. Now, how much will it cost us, by the way? Every minute. Three days left on your trial. Three days left on our trial, yes. I'm getting nervous here. So hang on. Per minute, we're paying $0.0015. $0.0015.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
which means that for 40 minutes, like, okay, for an hour, we're paying less than 10 cents for an hour of build time. So, you know, pay as you go. It's really not expensive. So it's okay if I have to put my card because we're talking cents per month for our builds. That makes sense. What does a single build cost us then? So when it's five minutes, let's see, I'll do the math now really quick.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Hang on. Thank you. Hang on. It's less than a cent. A build which takes five minutes is less than a cent. Is that right? Yeah, less than a cent, like 75. What is less than a cent? Zero cents. No, there was like another unit in the past. I forget what it's called. Whatever, like it's- Satoshi? No, that's a different- The 70, 75% of a cent. Okay.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
I like the idea of having this 20 line varnish config that we deploy around the world. And it's like, look at our CDN guys. It's so simple, and it can do exactly what we want it to do and nothing more. But understand that that's a pipe dream. That's where the name came from.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Okay, so it's like definitely- So reasonable, that's reasonable. Yeah, very, very reasonable, I would say.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
So namespace, basically it runs custom GitHub actions. much quicker because they have better hardware, better networking than GitHub Actions themselves. So you can literally use namespace to replace GitHub Actions.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
smart or you can use like faster docker builds you know but they also have preview environments which i haven't tried in code sandboxes that's something sponsor i that's what i'm thinking because i have a shout out here and hang on let me just get the name straight to be clear they are not a sponsor but we're saying they should i think they should be i think uh hugo i just know his first name and i'm trying to find um because our credit card is expiring
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Right, we need those six cents, don't we? We need those six cents. That's Gerhard's credit card for now. Exactly, you can use mine, it's okay. Hugo Santos. No relation, no relation. Yeah, no relation, no relation, but I think if there is someone that you should talk at Namespace, I think it would be him.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
As I was setting all this stuff up, he was very responsive even on the weekend to emails, and I think he's one of the founders, by the way, so I thought that was a very nice touch. And he really helped like go through all like the various questions which I had and the various like, does this look right? So even like he even looked at the pull request to see how we implemented it.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
And all in all, like the promise is there. We can see that it does work well when it works like two minutes, we get those two minutes, but sometimes it takes more. And then the question is, well, why does it take more? So that's something which I'm going to follow up on.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Cool. Cool. Well, I'm excited for the follow-up and for this progress. Indeed. Cool.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Because the varnish config will be slightly longer than 20 lines, and we'd run into all sorts of issues that we end up sinking all kinds of time into. Jared Santo, March 29th, 2024. Change it on with friends, episode 38.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
So I would like to switch gears to one of Adam's questions. And he was asking if Neon is working for us as expected and the state of Neon. So is Neon working for us as expected? Based on everything I've seen, it is. Like I was looking at, for example, the metrics. I was looking at how it behaves in the Neon console. This is us for the last 14 days.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
So what we see here in the Neon console, we see our main database. we can see that we have been using 0.04% of a CPU. So really not CPU, but in terms of memory, we have eight allocated. We're using 1.3 gigabytes. 1.3 gigabytes used out of eight allocated. So we are over-allocating both CPU and memory. So fairly little load, I would say, and things are just humming along. So no issues whatsoever.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Yes, I was. I didn't get to that, but that would not use this database, by the way. That would be something different now.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Right. I would, but not in this production database. So this is special, right? I mean, this is exactly what we want to see. If anything, we can, because we have the minimum compute setting set to two CPUs and eight gigs of memory. And I know that Neon does an excellent job of auto-scaling us when we need to. We didn't need to get auto-scaled because we are below the minimum threshold.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
So we could maybe even lower the threshold and it would still be fine.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
So we need some arbitrary workloads in order to push it. Well, to see where it breaks, we wouldn't need it to break. I think if anything, one thing that I would like us to do more is use Neon for development databases. And I have something there I haven't finished, but I would like to move on to that as well, if everyone's fine.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Yes, that's coming. That's my next topic to share. What exactly is that? Well, that would mean tooling that's local to make all of this easy. Jared wouldn't need to go to the UI to click any buttons to do anything. He would just run a command locally and the thing would happen locally. He wouldn't need to even open this UI. Shouldn't that be a Neon native thing though? It is.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
It does have a CLI, but the problem is you need to, first of all, install the CLI, configure the CLI, like add quite a few flags, connect the CLI to your local Postgres, like all that glue. That's the stuff which I've been working on. And I can talk about that a bit more.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Some command that you run locally that knows what the sequence of the steps is and what to do. For example, maybe you don't have the CLI installed. Well, install the CLI. you need to have some secrets. Well, here's the one password CLI. And by the way, the secrets is here, like in this vault. So stuff like that, like all that. Yeah.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
So it's possible to use something else. But at the end of the day, it's like the integration needs to work. And the implementation, whether you use the SDK or whether you use the CLI, is just an implementation. Just doesn't matter, yeah. What we care about is like, is it reliable, our implementation? Do we have any issues with it? So far, no. Yeah. Are we using like service accounts?
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
And that's something that we've been waiting because without service accounts, you would need to set up a connect server, which I didn't want to do. So that was a big deal for us. Whether you use the CLI or the SDK, we could, but it wouldn't make that much of a difference. Now, if the application itself, while it runs, it was doing certain things,
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Maybe that's interesting, maybe we could change some of the boot phase so that we wouldn't inject the secrets from outside the application and the application itself could get them directly. But I really want to get Elixir releases going. And once we have those, things change a little bit.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
But it's all just like maybe shuffling some code from here to here, but ultimately it will still behave the same, you know, just like you would maybe bring it into the language. So I haven't seen their latest SDKs, but I would like to check them out. That's a good one for me to look into. Okay.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
So the tooling that Jared was mentioning to make things simpler for him, I've been thinking about it from a couple of perspectives. And I realized that to do this right, it will be slightly harder. And the reason why it's slightly harder is because I would like to challenge a status quo. The status quo is you need a dagger for all of this. Maybe you don't, right?
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
So I'm trying a different approach. And the mindset that I have for this is Ken Beck, September, 2012. For each desired change, make the change easy. Warning, this may be hard. Then make the easy change. So what I've done for this Kaizen, I made that change easy, which was hard, so that I could make the easy change. How hard was it? So that's what happened. Well, let's have a look at it.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Yeah, I mean, the first, the initial commit of the repo was basically I extracted what would have become a pull request to the changelog repo. That was initial commit and we ended up with 46 lines of varnish config. The pull request won, which added dynamic backends. And it does something interesting with a cache status header. We end up with 60 lines of varnish config. Why dynamic backends?
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
So we're looking now at pull request 521. And 521 introduces some new tooling. But I promise it's just a CLI. And what's special about it is that everything runs locally. There's no containers. There's no Docker. There's no Dagger. Everything is local. And I can see Jared's eyebrows go up a bit because that's exactly what he wanted all this time.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
So what pull request five to one introduces is just. which is a command runner. It's written in Rust, but it's just a CLI. And if you were, for example, Jared, or even Adam could try this, if you were to run just, in our repository at the top level, you would see, Just calls them recipes, what is possible. And the one which I think the audience will appreciate is Just Contribute.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
So remember how we had like this manual step, like install Postgres, you know, get Erlang, get Elixir, get this, get that. I mean, that's still valid, right? You can still use that manual approach. Or if you run Just Contribute, it will do all those things for you running local commands. It still uses Homebrew, it still uses ASDF, but everything that runs, it runs it locally.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
And the reason why this is cool is because, I mean, your local machine, whatever you have running, it remains king. There's no containers. Again, I keep mentioning this because that adds an extra layer. And what that means, stuff like, for example, importing a database in a local PostgreSQL is simpler because that's what you already have running.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Resolving the Neon CLI, again, it's just like a brew install. It's there and you wire things together. You don't have to deal with networking between containers. You don't have to pass context inside of containers, which can be tricky, especially when it comes to sockets, especially when it comes to special files. So I'm wondering, how will this work out in practice?
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
And the thing which I didn't have time to do, I didn't have time to implement just db-prod-import, which would be the only command that you'd run to connect to Neon, pull down whatever needs to pull down, maybe install the CLI if it doesn't have it, and then just in your local Postgres, import the latest copy. Same thing for just dbfork, which would be an equivalent of what we had before.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
The difference is that was all using Dagger and containers and, you know, it was, I mean, have you used it, Jared, apart from when we've done it?
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
There you go. Adam, have you ever run Dagger? Never. In the three years that we've had it? Never. Not one time. There you go. How many times did you have to install things locally for you to be able to develop changelog in the last three years?
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
So Adam might be more excited about this than I am. Pull request 5 to 1. I mean, even you, Jared, if you want to try it. I mean, if you do dry run, it has a dry run option, by the way. It won't apply anything, but it will show you all the commands that would run if you were to run them yourself, for example. And there may be quite a lot of stuff, right, when you look at it that way.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
But it's a good way to understand, like, if you were to do this locally, and if you were to configure all these things, what would it do without actually doing it? So I tried it on a brand new Mac, and I think that's the recording that I have on that pull request.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
That's very, very... I've been waiting for a good reason to upgrade, you know? There you go. And honestly, within five minutes, depending on your internet connection, everything should be set up. Everything is local, the Postgres, everything.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
What we don't yet have, and I think this is where we're working towards, is how do we, first of all, cleanse the data so that contributors can load a type of data locally. But I think that's like a follow-up. First of all, we want Jared to be able to do this with a single command, refresh his local data. And after I have this, the bulk of the work done, this step is really simple. How simple?
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
That was an important one because whenever there's a new application deployment, you can't have static backends. The IP will change. Therefore, you need to use the DNS to resolve whatever the domain is pointing to. So that's what the first pull request was. And that's what we did in the second iteration. Now, I captured what I think is a roadmap. It's in the repo.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Maybe half an hour at most. That's what I'm thinking. So not much.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
I'm just curious about that change. so i mentioned it in one of the comments when i committed basically when i was installing it via asdf the problem with was with icu4c i just couldn't compile postgres from asdf correctly and since then in homebrew we can now install postgres at 16. so you can specify which major version which was not possible i think two years ago when i did this initially
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
so there is that now let's see let's see where this goes i'm excited about this if anyone tries it let us know how it goes for you uh if you want to contribute a changelog like how far does it get and by the way i tested this on linux as well the easiest way there's like a something hidden there in the just it's called actions runner what it does is exactly what you think it does it runs
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
a GitHub actions runner locally, for this you need Docker by the way, and it loads all the context of the repository inside of that runner. So that's the beginning of what it would take to reuse this in the context of GitHub actions. And what I'm wondering is, will it be faster than if we use Dagger? That's me challenging the status quo. The answer is either, yes, it is.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
And maybe we should do that instead. It will shave off more time or no, it's not. And then I get to finally upgrade Dagger because we're on a really old version. So you still work at Dagger, right? I do. Yes, very much so. Yes. Okay.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
No, no, we do actually run the commands. It's going to do it for you, man.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
What you could do is, if you want, maybe start a new user. It shouldn't mess anything up, to be honest. It just installs. Maybe it does things differently or does things twice. I don't really know. But it should be safe.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
And it's all local. That's the big difference. Because before, I mean, even now, right, because we still kept Dagger, we still have everything that we had so far, that would always run in containers, which means it wouldn't change anything locally. And in some cases, that's exactly what you want, especially when you want to reduce the parity between test and production or staging and production.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
But in this case, it's local, right? So you want something to happen locally. And local is not Linux. It means it's a Mac. So then you have that thing to deal with. in which case brew helps, and ASDF helps, and a couple of tools help. But you still have to know what are the commands that you have to run, in what order, what needs to be present, when. And this basically captures all those commands.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
And I was going to ask you, what do you think about the idea in terms of what's coming? So the next step would be to add the feeds backend. Why? Because feeds, we are publishing them to Cloudflare R2. So we would need to proxy to that, basically cache those. I think that would be like a good next step.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
It's a little bit like make, which we had, and we removed. But this is a modern, I would say, version of that. Much simpler, much more streamlined, and a huge community around it. I was surprised to see how many people use Just. By the way, huge shout out to Casey, the author of Just. I really like what he did with the tool, like 20,000 stars on GitHub.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
A lot of releases, 114 fresh releases, 170 contributors. Yeah, it's a big ecosystem, I have to say. Mm-hmm.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Exactly. Okay. And dash n, it basically stands for dry run.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
The reason why you have to do it before the recipe is because some recipes can have arguments, and if they don't, like if you do the dash n at the end, it won't work. So it has to be the command just, the flags, and then the recipe or recipes because you can run multiple at once.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Casey. Let me see if I can pronounce his surname. He's Casey on GitHub, by the way. Rodarmore? The blue planet, apparently. Casey Rodarmore. You can correct us, Casey.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Rod armor. Rod armor. Rod armor. Yes, I like that. That's how we're pronouncing it.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Not a problem. I enjoyed it. It was fun. Okay. Homelab production. Homelab to production. So next week on Wednesday, it's TalosCon and I'm calling it Justin's conference. It's the Garrison Con. The Garrison Con, exactly. I'll finally meet Justin in person. I'm giving a talk. It's called Home Lab to Production. It's, I think it's 5 p.m. So one of the last ones. We'll have a lot of fun.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
I'm bringing my home lab to this conference. So we will have fun.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
It is a mobile app, but I will have a router with me. So it will be both the actual device and the router. And yeah, we'll have some fun. Now you're bringing two of them with you or just one? The device, the Homelab plus the router. So two devices. Okay. I want two of everything. Yes. Well, we are going into production.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
So we're going to take all the workloads from the Homelab and we're going to ship them into production. During the talk, we're going to see how they work. We're going to use Talos, Kubernetes, Dagger is going to be there. So yeah, we'll have some fun. So this is a live demo then, basically. It's a live, yes.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Well, it's recorded because, you know, I want to make sure that things will work, but I will have the devices there with me. You never know what Wi-Fi is like. And that's the one thing which I don't want to risk.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Even like 4G, 5G, even mobile networks are sometimes unreliable. But I'm looking forward to that. So that's like, and it will be a recorded talk as well. So yeah.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
then i'm thinking we should figure out how to send the logs to honeycomb exactly the same as we currently send them so that you know same structure same dashboard same query same slos everything that we have configured in honeycomb would work exactly the same with the new logs from this new cdn Then we need to do implement the purging across all instances.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
tense so there you go and guess what what I'm going to be recording myself as well okay what are you holding up there I'm holding a Rode Pro. Do you know the Rode Pros? Like the mini recording microphones?
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Exactly. So I have two of those. Boom. And two cameras. I'll take them with me. They're 361. So I'll be recording like the whole talk and then editing and publishing it. So that's the plan. Cool. So whatever the conference does, great. But I also want to do my own.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
So that's the plan. Full control.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
What do we want to accomplish for the next one? Are we on the right trajectory? Like in terms of the things that we talked about, in terms of what we think is coming next, did we miss anything? It'll be Christmas or just before Christmas.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Pipely.tech. I like it. Did you buy the domain?
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
But I like pipe.ly.tech as well.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
I think that's slightly harder because as we deploy the CDN in like 16 regions, 16 locations, we would need to expire, right? Like when there's an update. So that I think is slightly harder, but not crazy difficult. And then we would need to import all the current edge redirects from our current CDN into the pipe dream. And I think with that, we could try running it in production, I think.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
So we can still keep S3, whatever intercepts the logs, right? Because in our current CDN, obviously the CD intercepts all the logs. And then some of those logs, they get sent to S3 indeed. But then all the logs, they get sent to Honeycomb. So you're right, I forgot about the S3 part.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
So on top of sending everything to Honeycomb, we would also need to send a subset to S3 exactly as the current config. So yes, that's an extra item that's missing on that roadmap indeed.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
No idea currently. I mean, based on our architecture and what we have running so that we avoid introducing something new as a new component, a new service that does this, we could potentially do it as a job using OBAN, I think. Because at the end of the day, it's just hitting some endpoints, HTTP endpoints, and it just needs to present a key, right?
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
If we don't use it, anyone can expire our cache, which is a default in some CDNs. Yeah, it is. Yeah, we found that out the hard way. Exactly. So that's something that we need. I think an O-band job would make most sense. It's actually pretty straightforward.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Yeah. We can get that information by doing a DNS query and it tells us all instances and then we can get all the URLs.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
So Pipedream is our own CDN, which caches requests going to backends. So imagine that there's a request that needs to hit the app and then the app needs to respond. So the first time, like let's say the home page, once the app does that, subsequent requests, they no longer need to go to the app. Pipedream can just serve because it already has that request cached.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
And then because Pipedream is distributed across the whole world, it can serve from the closest location to the user. To the person. Exactly. And same would be true, for example, for feeds, even though they are stored in Cloudflare R2. The PipeDream instance now goes to Cloudflare R2, gets the feed, and then serves the feed.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
So by default, we're using memory, but using the static backend like a disk backend would be possible, yes.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
I don't know. I quite like the name, to be honest. I think it has a great story behind it, you know? So it just goes back to the origin.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
Yeah. Yeah, I like that pipe idea. That was like one of the follow-up questions. Do we keep a space or introduce a space? Or no space? That's a really important decision. Space or no space? What about a tab? Should we put a tab in there? We can.
The Changelog: Software Development, Open Source
Kaizen! Just do it (Friends)
What do listeners think? I mean, you've been hearing this story for a while and you've heard us think. I think we should have a poll. And that's how, you know, I know that's how we end with names like Boaty McBoatface. We're very aware of that. This is not that. We're just asking, like, how do we, what would be the way to spell it that would make most sense?